Modeling and Results Supplement

Purpose of the Document

The aim of these supplementary details are to provide readers with better documentation of the complete modeling and model selection process. There were a total of 11 models fit as part of this study, and the number of analyses and supporting figures to understand this process are simply too long for a single document. This document therefore provides relevant plots and summaries to help readers understand the steps taken to arrive at the final model, and perhaps more importantly, the document provides commentary for how each modeling choice was made.

The goal of this document is therefore to ensure complete transparency in the data analysis process. Accompanying this document is also the R script file that was created during the study process. Someone who acquires access to the appropriate files through the HRS should be able to run the script file (with some minimal alterations to ensure that the files are read from the correct files) and then get results that match those shown here (within some margin of error due to randomness in some of the methods, though the use of the same seed in R and brms should reduce the impact of this).

Overview of Sections

In an attempt to facilitate readability of this document, the modeling process is broken down into a few different sections. Additionally, there is a table of contents that readers can use to jump to any section, so for those interested in specific aspects of this supplementary material, it is useful to know what exists in each section.

First, a flowchart showing the order in which models were built and the model to which each was tested is provided. Each candidate model was compared to the up-to-then best fitting model. The aim of this flowchart is to help familiarize readers with the models and modeling steps quickly and efficiently as the remaining sections add greater detail.

Second, the priors used in the models are explored. As the data analyses were done in the Bayesian framework, the analysis of the priors is useful. The priors are shown as well as the prior predictive checks for a fixed item model.

Third, the details of each model are provided. This is the largest section as each model includes a variety of explorations. To help reduce the overall length of the document, each model is given its own tab so that readers can select one model at a time. Details for each model can be broken down into 3 primary sections: model validity, model performance, and model estimates.

  • Model validity refers to tests of whether the estimation process converged and not subject to any issues that would make estimates from the model entirely invalid or unstable. These tests include visual inspection of chain mixtures, \(\hat{R}\), effective sample size, and maximum treedepth.
  • Model performance refers to the posterior predictive checks that mirror those shown in the manuscript for the final model: predictions of all responses for all items, responses to each item, and responses to all items for a subset of participants. As the space of this document is less limited than space in the final manuscript, the random subset of participants is increased from 12 to 20.
  • Model estimates refers to the summary of model parameters like fixed and random effects estimates. This summary is different than the one presented in the manuscript for the final paper as the objective of these intermediary models is not to summarize effect sizes or the probability of these effects; instead, the goal is to get a general idea of what the model is estimating and how it is performing. Toward this end, conditional effects plots for each model are also included. Note that these plots may not be very informative for the majority of the models tested because only a subset included many covariates.

Finally, as discussed in the manuscript, some additional details regarding the final model are also included here. These details include summaries of the item parameters in traditional IRT metrics (i.e., as difficulty and discrimination), the reliability plot, expected score functions (total test), and information functions (total test and all trials). As in the manuscript, these item-specific plots are provided with and without assuming previous learning of the words. The test is best understood as a dynamic one in which a person’s performance on each trial changes our expectation for how they will perform on the next.

What’s Being Done

As mentioned earlier, this document shows all the code used to generate the results. Since there is an accompanying R data script, it may be useful for readers to know the objects being called in this Markdown document since those objects can be connected back to the R script. The hope is that this will create a reasonable sense of cohesion between the supplementary materials, and it should mean that all the results here are also fully reproducible. Toward that end, the objects and packages used in this document are shown below (note that the R objects are read in as RDS files whose names are consistent with those listed in the R script file).

#read in needed data
df_long <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Saved Objects for RMDs/Item Covariate Analyses/Data_long.rds")
Rasch_prior <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/1PL_prior_check.rds")
TwoPL_prior <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_prior_check.rds")
Rasch_inter <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/1PL_intercept.rds")
TwoPL_inter <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_intercept.rds")
TwoPL_learn <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_growthModel.rds")
TwoPL_multi <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_multidimensional.rds")
TwoPL_chnge <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_changeModel.rds")
TwoPL_depmd <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_dependencyModel.rds")
TwoPL_depun <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_dependencyUniqueModel.rds")
TwoPL_deptr <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_dependencyTrialModel.rds")
TwoPL_itmex <- readRDS("C:\\Users/billy/Desktop/Psych Articles/HCAP/CERAD IRT Update/R Files/Fitted Models/Item Covariate Models/Revised Models/2PL_itemCovariates.rds")

#load in required packages 
#wrapping in suppressPackageStartupMessages() done just to reduce print out in document
suppressPackageStartupMessages(library(brms))
suppressPackageStartupMessages(library(tidyverse))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(kableExtra))

Modeling Flowchart

As can be inferred from the figure below, the modeling process involved several iterations of fitting similar models and considering results from old models. As there specific hypotheses regarding the model and its covariates, the models needed to test these hypotheses were naturally created as part of the study. At the same time, alternative models had to be specified against which to test these models. Additionally, while there were hypotheses regarding what covariates (and their signs) would ultimately be in the final model, the important result of the study is the final model itself. Some readers may see the number of models examined and variations in their specification and become concerned of potentially two things: multiple comparisons + inflated error rates and/or model fishing/p-hacking.

The objective of this supporting document is to help clarify all the modeling choices so that readers do not need to question whether model specifications were made to try to improve performance of the final result. With respect to possible concerns regarding multiple comparisons, Bayesian methods do no suffer from these concerns (Gelman et al., 2013; Gelman, Hill,& Yajima, 2012; Neath, Flores, & Cavanaugh, 2017; Sjölander & Vansteelandt, 2019). While there are several reasons that this is the case for Bayesian methods, it is sufficient to speak to three. First, we do not use null hypothesis testing in this study. Model comparisons are completed using a formal comparison of information criteria to select models with better out-of-sample performance. Coefficients are not interpreted as significant or not but instead are summarized in terms of their probability of existing. Since the correction to p-values for multiple comparisons is to control the risk of falsely rejecting the null hypothesis, this is not a concern when we are not rejecting null hypotheses. Second, we utilize skeptical priors for our effects estimates. This means that we are a priori placing greater probability of the effects being 0 (or practically equivalent to 0). This is the inverse of frequentist decision-making practices where the null hypothesis is very easy to reject since it is constrained to (usually) a nill point value, which is rarely a realistic threshold for any model specification. Finally, the models include skeptical priors, meaning that all effects estimates are pulled closer to those a priori small effects.

Flowchart of all models fit and compared in order (click to make larger)

Prior Specifications and Inspection

As discussed in the manuscript for this study, prior specification came from documentation on using brms for IRT (i.e., Bürkner, 2020a and Bürkner, 2020b). As a general note, the non-linear specification for the 2PL model used in this study come from the Bürkner (2020b) study published in the Journal of Intelligence. Also as discussed in the manuscript, the specification of priors follows the recommendations of other typical multilevel regression guides (e.g., Gelman & Hill, 2007). Specifically, the priors are normal distributions with wide variances relative to the scale of the outcome data. As these priors on the regression coefficients, normal distributions are appropriate prior distributions. While these distributions are centered on 0, they are made wide and thus only weakly informative to the final parameter estimates. This specification helps regularize estimation (i.e., pull estimates toward zero and away from more extreme values) while imparting little a priori influence on the estimates. Additionally, by making the priors skeptical (i.e., they place the greatest probability on very small to non-existent effects), there is a reduction in the risk of experimenter bias; however, with 1219 participants each observed 30 times, the data will dominate the prior anyway.

The priors for the Rasch and 2PL models are shown below:

Rasch_priors <-
  prior("normal(0, 5)", class = "b") +
  prior("normal(0, 3)", class = "sd", group = "ID")

TwoPL_priors <- 
  prior("normal(0, 5)", class = "b", nlpar = "beta") +
  prior("normal(0, 1)", class = "b", nlpar = "logalpha") +
  prior("normal(0, 1)", class = "sd", group = "ID", nlpar = "theta")

Readers following the R script file will recognize that the above are repeated in that document (“3 - Measurement and Explanatory Model Fit”, lines 19-27). To read these priors, it can be helpful to look at a couple of examples. Starting with the priors for the Rasch model, the prior for the coefficients (class = "b") is specified as a normal distribution with a mean of zero and standard deviation of five. This would mean that, before looking at the data, we are guessing that there is a 68% probability of the intercept being between -5 and 5 (i.e., +/- 1SD), and we are guessing that there is about a 95% probability that the coefficients will be between -10 and 10 (i.e., +/- 2SD). In these models, the coefficient corresponds to the actual item parameters (for the Rasch, this is item easiness), so these are the ranges in which we are saying are potential values we might observe. Take another example but this time from the 2PL model and a random effect. The random effect priors are all labeled as class = "sd" since we are putting a prior belief on the plausible values of the standard deviation of the random effects. For the random person effect (i.e., latent trait of each participant), we look for the variable that defines this group (group = "ID", where ID is a number indexing each participant) and the non-linear element it is estimating (nlpar = "theta" where \(\theta\) is the traditional IRT simple for the latent trait). The prior therefore for the latent trait is the standard normal distribution with mean of zero and standard deviation of 1. This specification is consistent with the treatment of the latent trait in IRT as normally distributed and z-scaled, though generally in IRT models the variance is constrained to be 1 for identifiability purposes (see Bürnker, 2020a for details).

While it’s comforting to be able to go through each prior specification and think about what it means, it is perhaps more efficient to examine some plots. The first set of plots shown are the parameter estimates returned when the model samples only from the prior. In other words, these are the estimated effects implied by the priors. If the prior specifications are truly skeptical and weakly informative, then they will give the greatest probability to effects of very small size and cover a wide range of plausible values. The prior distribution for the Rasch easiness parameter of “Butter” on trial one.

plot(Rasch_prior, combo = c("dens", "intervals"), variable = "b_ItemButter1", ask = FALSE)

Consistent with expectations, these plots show wide ranges of plausible values with the greatest probabilities being placed on small effects. In the left column are the density plots of the estimates while the right column shows the interval estimates with the circle = mean estimate, bold line = 50% credible interval, and thin line = 90% credible interval. The prior density very clearly follows a normal distribution that allows admissible values with more extreme estimates being given small overall probability of being true. These wide priors allow the data to dominate the posterior estimates for the parameters, though again this would likely be the case with even more informative priors due to the size of the sample available.

plot(TwoPL_prior, combo = c("dens", "intervals"), variable = "b_beta_ItemButter1", ask = FALSE)

The plot above mirrors what was shown for the Rasch model but are for the 2PL model, which is clearly not different as the same priors are used for this parameter. Though all the prior distributions for the model can be shown, this results in many different plots that can be overwhelming to interpret, and these plots correspond to just priors for those specific parameters. Since researchers may not often have explicit expectations for what each parameter will be or should be, these plots are not always directly helpful.

Another related graphical output is the prior predictive check. The prior predictive check runs the model using the priors rather than the observed data (note that this model would be similar to a null model). If the priors are specified well, then they should return reasonable, albeit wide, estimates of the observed data. This plot is more intuitive for understanding the effect of priors instead of parameter-by-parameter plots like above. The prior predictive checks for the Rasch and 2PL models are shown below following the same layout as the posterior predictive checks in the manuscript and for the other models.

pp_check(Rasch_prior, ndraws = 25, type = "bars")

pp_check(Rasch_prior, ndraws = 25, type = "bars_grouped", group = "Item")

pp_check(Rasch_prior, ndraws = 25, type = "bars_grouped", group = "ID", 
         newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))

The Rasch prior predictive checks above demonstrate that the prior specifications are adequately wide to provide coverage for the observed data. The estimates themselves are expectedly poor, which is a product of the skeptical priors. The same plots are now repeated but for the 2PL model.

pp_check(TwoPL_prior, ndraws = 25, type = "bars")

pp_check(TwoPL_prior, ndraws = 25, type = "bars_grouped", group = "Item")

pp_check(TwoPL_prior, ndraws = 25, type = "bars_grouped", group = "ID", 
         newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))

Performance of the 2PL priors are similar to those of the Rasch, suggesting that these priors are also appropriately specified.

While all the above plots and theoretical justifications suggest that the priors are specified consistent with the wishes for the model, it can also be helpful to perform a post-hoc test of whether a model’s priors were influential on its final estimates. As has been mentioned multiple times in this section, due to the sample size of this study, it is expected that the data and not the prior will dominate the posterior estimates, meaning that even with more informative priors the data would have more influence on the final estimates. One such comparison discussed by Andrew Gelman (link here) is to compare the posterior standard deviation (i.e., precision of the effect estimate after looking at the data) to the prior standard deviation (i.e., uncertainty of the effect estimate before looking at the data). In the case that a prior is influential, the ratio of the precision to uncertainty will be large. Put another way, we learn little more about the posterior from observing the data because the prior was already highly informative. Gelman’s recommended threshold for determining whether a prior is informative is if the posterior standard deviation for an effect is more than 0.1 times the prior standard deviation. The able below provides this metric for each predictor from the final model reported in the study.

#get the posterior samples from the final model
posteriors <- as_draws_df(TwoPL_itmex)

#get the fixed effects for the item easiness
beta <- posteriors %>%
  select(starts_with("b_beta")) %>%
  apply(., 2, function(x) sd(x)/sd(posteriors$prior_b_beta)) %>%
  as.matrix()
## Warning: Dropping 'draws_df' class as required metadata was removed.
#do the same for item discrimination
alpha <- posteriors %>%
  select(starts_with("b_logalpha")) %>%
  apply(., 2, function(x) sd(x)/sd(posteriors$prior_b_logalpha)) %>%
  as.matrix()
## Warning: Dropping 'draws_df' class as required metadata was removed.
#combine into a single result
result <- rbind(beta, alpha) %>%
  as.data.frame() %>%
  add_column("Prior Influence" = ifelse(.[, 1] >= 0.1, "Informative", "Uninformative")) %>%
  rename("Ratio (Posterior:Prior)" = V1)
row.names(result) <- paste(rep(c("Easiness:", "Discrimination:"), each = 11), rep(c("Intercept", "Trial 2", "Trial 3", "Item Pos. (linear)", "Item Pos. (quadratic)", "Word Frequency", "Concreteness", "Semantic Diversity", "Age of Acquisition", "Body-Object Integration", "Phonemes"), 2))
rm(posteriors, beta, alpha)

#get resulting table
result %>%
  kable(caption = "Comparison of the Posterior to Prior Distribution Standard Deviations", digits = 4, align = 'cc') %>%
  column_spec(1:3, bold = ifelse(result$`Ratio (Posterior:Prior)` >= 0.10, TRUE, FALSE)) %>%
  kable_classic(full_width = FALSE, position = "float_right")
Comparison of the Posterior to Prior Distribution Standard Deviations
Ratio (Posterior:Prior) Prior Influence
Easiness: Intercept 0.0122 Uninformative
Easiness: Trial 2 0.0146 Uninformative
Easiness: Trial 3 0.0153 Uninformative
Easiness: Item Pos. (linear) 0.7991 Informative
Easiness: Item Pos. (quadratic) 0.7709 Informative
Easiness: Word Frequency 0.0088 Uninformative
Easiness: Concreteness 0.0125 Uninformative
Easiness: Semantic Diversity 0.0125 Uninformative
Easiness: Age of Acquisition 0.0105 Uninformative
Easiness: Body-Object Integration 0.0127 Uninformative
Easiness: Phonemes 0.0093 Uninformative
Discrimination: Intercept 0.6915 Informative
Discrimination: Trial 2 0.0628 Uninformative
Discrimination: Trial 3 0.0662 Uninformative
Discrimination: Item Pos. (linear) 0.9746 Informative
Discrimination: Item Pos. (quadratic) 0.9916 Informative
Discrimination: Word Frequency 0.0407 Uninformative
Discrimination: Concreteness 0.0575 Uninformative
Discrimination: Semantic Diversity 0.0586 Uninformative
Discrimination: Age of Acquisition 0.0532 Uninformative
Discrimination: Body-Object Integration 0.0515 Uninformative
Discrimination: Phonemes 0.0459 Uninformative

The table of these posterior and prior comparison results is shown here. For convenience, only those whose threshold exceeds the recommended > 0.10 ratio are bolded. Generally, the findings suggest that the priors performed as expected: they were weakly informative and did not seemingly have undue influence on the posterior estimates. Notably, the exceptions are the discrimination intercept, which had a 95% credible interval that included zero, and the item position coefficients that had relatively larger standard errors. This is not unexpected as the result essentially indicates that even after looking at the data we do not change our prior beliefs, which were that the effects were either zero or very wide. The influence of the prior on these null findings reflects a point made earlier regarding how Bayesian methods are generally unaffected by multiple comparisons.

An important point to emphasize at this juncture is the implication of a “significant” finding in Bayesian methods. As discussed throughout this section on priors, the priors here are skeptical of an effect in the sense that they place greatest weight on an effect estimate of zero or close to zero and ambivalence regarding the direction of the effect (i.e., it is just as equally probable that the sign is positive of negative). In the context of the current study, this means that, despite the hypotheses regarding the presence and direction of specific effects, the priors for these predictors are specified in this skeptical way so as to avoid the introduction of experimenter bias. In regard to the robustness of the effects observed, the fact that they are observed from the information provided by the data despite these skeptical priors also helps build confidence in the presence of these effects.

Model Details

The details of this section highlight the models fitting and results. These details speak to the validity of the model results and then also the actual results (i.e., parameter estimates) of the model. Model validity is particularly important in Bayesian methods because the parameter estimates are based on Monte Carlo Markov Chains (or Hamiltonian Monte Carlo (HMC) in the case of these models run using Stan). In cases where a model fails to converge or throw errors under the estimator, the validity of the model results are questionable or even completely invalid (e.g., in the case of divergent transitions). To reflect this need to first confirm the validity of the results, various diagnostics of the model fit are provided first before then presenting the model results.

For readers unfamiliar with these model checks, a brief overview of each is provided here. The largest threat to model results in the HMC is arguably the presence of divergent transitions. HMC explores the posterior distribution by simulating the evolution of a Hamiltonian system, and in order to do this efficiently, the sampler finds a reasonable step size with which to explore that space. A divergent transition occurs when the trajectory of the system is lost due to too large of a step size. Another important model check is the treedepth of the chains. Again, to improve the efficiency of the posterior sampling, a maximum treedepth is set to prevent the estimator spending excessive time in certain steps and spaces. Since this treedepth may artificially limit the estimator in exploring the posterior space, it is important to check whether any of these treedepths were actually hit during estimation (default treedepth is 10). Another important Bayesian model indicator is \(\hat{R}\) because multiple HMC (and MCMC) chains are needed to ensure that the posterior is sampled appropriately. If a single chain is run, then it is not possible to determine whether the random starting values of this chain may have lead to a specific set of parameter estimates. Running multiple independent chains that each have different random starting values helps ensure that the parameter estimates are not biased by exploration of only certain posterior values. In well-behaved models, these chains will mix together without any clear indications of one chain producing a specific set of parameter estimates that differ from what other chains are estimating. While this mixture of chains can be visually inspected via the trace plot (also provided here), the \(\hat{R}\) statistic is a simple indicator of this with the conservative recommendation of treating estimates as valid only if the \(\hat{R}\) for the parameter is less than 1.01. A final model validity check shown here is the effective sample size. Because multiple chains are run for many samples of the posterior, it is expected that some of those samples are autocorrelated and thus dependent on previous samples. Effective sample size informs us of the precision of the model in MCMC and HMC methods. When samples are independent, the central limit theorem indicates that the precision with which a parameter can be estimated is proportional to the size of the sample (e.g., \(\sigma_\bar{x} = \frac{\sigma}{\sqrt{N}}\)). The same proportionality can be obtained when samples are dependent but requires replacing \(N\) with \(N_{ESS}\), or the effective sample size. Due to the dependence of the sampling, \(N_{ESS} < N\) and thus precision of the estimate is less than would be if it could be estimated from the total sample. Running the chains for more iterations will necessarily increase \(N_{ESS}\), but there is a practical demand on computational effort relative to marginal increases in precision. The recommendation of the Stan developers is to run enough iterations of the sampler to obtain an \(N_{ESS} >= 4*N_{chains}\). All models were run using 4 independent chains, so the minimally acceptable ESS is 400 (i.e, 4*100).

In the case that the model checks are appropriate, it is appropriate to examine the posterior distribution and begin inference based on the results. While there are more parameters estimated by the models, posterior summaries of the coefficients in each model are shown. The posterior is summarized as a density plot that reflects the probability distribution of the parameter based on integrating our prior knowledge and observed data. The density plot shows the 95% credible interval with the 80% credible interval shown as a shaded area. Unlike frequentist confidence intervals, these credible intervals can be interpreted as the probability of the parameter having a specific value. For example, if the 95% credible ranges from 0.50 to 1.00, then this means that there is a probability of 0.95 that the parameter has a value somewhere within this interval. This is in contrast to frequentist confidence intervals where the same interval would be interpreted as meaning that 95% of point estimates based on the same statistical test applied to an infinite number of random samples of the same population will be within this interval. Thus, where the credible interval directly summarizes the our beliefs about the parameter and our uncertainty about its true value, the confidence interval only reflects point estimates that we would expect to observe if the study and statistical methods were repeated an infinite number of times. Posterior predictive checks of the models are then also presented as they were for the final model in the corresponding manuscript. There is one additional model exploration plot provided in this section that has not been addressed before in this document: the conditional effects plot. As the predictors in these models are correlated and have their effects estimated on the logit scale, it can be challenging to look at the model estimates and understand the implication of these values in an intuitive manner. One way to address this is to visualize how the predicted outcome of the model changes as a function of each predictor while holding all other predictors constant (e.g., at their mean value). The resulting plot is the conditional effects plot. In the case of these models, this plot shows, for each predictor, what happens to the predicted probability of a correct response as the value of the predictor changes and all other model values are held constant. These plots are not realistic as it is not reasonable to assume that there exist words whose traits can vary only on one property at a time; however, they do provide a quick method of understanding the relative effect of each predictor by showing its linear trend as implied by the model. As a result, these plots should not be used for prediction or extrapolation in any regard; instead, if the goal is prediction of responses, then the entire model should be used, and extrapolation of these predictions to values not observed in this study should be avoided. These plots are simply to help contextualize the meaning of the effect estimate in the model.

Rasch Fixed Items

mcmc_plot(Rasch_inter, type = "nuts_divergence")

mcmc_plot(Rasch_inter, type = "nuts_treedepth")

mcmc_plot(Rasch_inter, type = "trace", variable = "b_Item", regex = TRUE)
## No divergences to plot.

mcmc_plot(Rasch_inter, type = "trace", variable = "sd_", regex = TRUE)
## No divergences to plot.

mcmc_plot(Rasch_inter, type = "rhat_hist", binwidth = 0.0001)

mcmc_plot(Rasch_inter, type = "neff_hist", binwidth = 0.1)

The fixed items Rasch model demonstrated no evidence of estimation concerns that would raise concerns for the validity of the results. As a result, we can look at the results from the model overall.

mcmc_plot(Rasch_inter, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_Item", regex = TRUE)

pp_check(Rasch_inter, ndraws = 50, type = "bars")

pp_check(Rasch_inter, ndraws = 50, type = "bars_grouped", group = "Item")

pp_check(Rasch_inter, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))

Even with no predictors and fewer parameters in the Rasch model, the model does very well in predicting responses. The density plot of the item easiness estimates demonstrates that most items are fairly easy. The following general model summary integrates basic model validity statistics and posterior summaries for additional parameters.

summary(Rasch_inter)
##  Family: bernoulli 
##   Links: mu = logit 
## Formula: Resp ~ 0 + Item + (1 | ID) 
##    Data: df_long (Number of observations: 36570) 
##   Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
##          total post-warmup draws = 8000
## 
## Group-Level Effects: 
## ~ID (Number of levels: 1219) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.62      0.02     0.58     0.66 1.00     2712     3729
## 
## Population-Level Effects: 
##             Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## ItemButter1     1.32      0.07     1.17     1.46 1.00     6354     5699
## ItemArm1       -0.09      0.06    -0.21     0.03 1.00     6080     5708
## ItemShore1     -0.50      0.06    -0.62    -0.37 1.00     6764     5636
## ItemLetter1    -1.06      0.07    -1.20    -0.93 1.00     7192     5616
## ItemQueen1     -0.78      0.07    -0.91    -0.65 1.00     7200     5584
## ItemCabin1     -0.89      0.07    -1.02    -0.76 1.00     6397     5668
## ItemPole1       0.37      0.06     0.24     0.49 1.00     5281     4896
## ItemTicket1    -0.49      0.06    -0.62    -0.37 1.00     6757     5259
## ItemGrass1      0.79      0.07     0.66     0.92 1.00     6149     5198
## ItemEngine1     0.79      0.06     0.67     0.92 1.00     6451     5835
## ItemTicket2     1.26      0.07     1.12     1.40 1.00     6541     5871
## ItemCabin2      1.08      0.07     0.94     1.21 1.00     6313     5562
## ItemButter2     0.29      0.06     0.17     0.42 1.00     6613     5404
## ItemShore2      0.33      0.06     0.21     0.46 1.00     6690     5771
## ItemEngine2     0.93      0.07     0.80     1.06 1.00     6462     5718
## ItemArm2        0.35      0.06     0.23     0.47 1.00     5263     4942
## ItemQueen2      2.08      0.09     1.91     2.25 1.00     7259     5478
## ItemLetter2     0.85      0.07     0.72     0.98 1.00     6304     5246
## ItemPole2       2.38      0.10     2.19     2.59 1.00     7903     5077
## ItemGrass2      0.23      0.06     0.10     0.36 1.00     5965     5110
## ItemQueen3      1.41      0.07     1.27     1.56 1.00     7296     5380
## ItemGrass3      1.45      0.07     1.30     1.59 1.00     6759     6069
## ItemArm3        1.10      0.07     0.97     1.24 1.00     7319     5392
## ItemCabin3      1.74      0.08     1.59     1.90 1.00     6984     5378
## ItemPole3       1.65      0.08     1.49     1.81 1.00     5876     5353
## ItemShore3      0.67      0.06     0.54     0.80 1.00     5905     5024
## ItemButter3     1.33      0.07     1.19     1.48 1.00     7179     5516
## ItemEngine3     1.89      0.08     1.73     2.05 1.00     7327     5954
## ItemTicket3     0.97      0.07     0.84     1.11 1.00     6798     5470
## ItemLetter3     0.67      0.06     0.54     0.80 1.00     5181     5429
## 
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

To help clarify the meaning of some major elements of the above output, consider the following guide: 1. “Estimate” refers to the average posterior value for the parameter, 2. “Est. Error” is the standard deviation of the posterior distribution, 3. “l-95% CI” is the lower bound of the 95% credible interval, 4. “u-95% CI” is the upper bound of the 95% credible interval, 5. “Rhat” is the \(\hat{R}\) value for that parameter (rounded to two decimal places), 6. “Bulk_ESS” is the effective sample size based on rank normalized draws and estimates the sampling efficiency of the mean of the posterior, and 7. “Tail_ESS” is the minimum of the effect sample sizes in the 5% and 95% quantiles.

2PL Fixed Items

mcmc_plot(TwoPL_inter, type = "nuts_divergence")

mcmc_plot(TwoPL_inter, type = "nuts_treedepth")

mcmc_plot(TwoPL_inter, type = "trace", variable = "b_beta_Item", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_inter, type = "trace", variable = "b_logalpha_Item", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_inter, type = "trace", variable = "sd_", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_inter, type = "rhat_hist", binwidth = 0.0001)

mcmc_plot(TwoPL_inter, type = "neff_hist", binwidth = 0.1)

The fixed item 2PL model demonstrated no evidence of estimation concerns that would raise concerns for the validity of the results. As a result, we can look at the results from the model overall.

mcmc_plot(TwoPL_inter, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta_Item", regex = TRUE)

mcmc_plot(TwoPL_inter, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha_Item", regex = TRUE)

pp_check(TwoPL_inter, ndraws = 50, type = "bars")

pp_check(TwoPL_inter, ndraws = 50, type = "bars_grouped", group = "Item")

pp_check(TwoPL_inter, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))

Much like with the Rasch model, the 2PL fixed item model does very well in predicting responses. There are two parameters (one for difficulty/beta and one for discrimination/alpha) being estimated since the 2PL model is a non-linear model. Note that the model’s name for the discrimination parameter is “logalpha.” This name reflects the fact that the alpha parameter was log-transformed to ensure that it was constrained in estimation to positive values. The following general model summary integrates basic model validity statistics and posterior summaries for additional parameters. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.

summary(TwoPL_inter)
##  Family: bernoulli 
##   Links: mu = logit 
## Formula: Resp ~ beta + exp(logalpha) * theta 
##          theta ~ 0 + (1 | ID)
##          beta ~ 0 + Item
##          logalpha ~ 0 + Item
##    Data: df_long (Number of observations: 36570) 
##   Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
##          total post-warmup draws = 8000
## 
## Group-Level Effects: 
## ~ID (Number of levels: 1219) 
##                     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(theta_Intercept)     0.51      0.10     0.35     0.72 1.00      556     1166
## 
## Population-Level Effects: 
##                      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_ItemButter1         1.53      0.10     1.34     1.73 1.00     5640
## beta_ItemArm1           -0.10      0.08    -0.26     0.06 1.00     3975
## beta_ItemShore1         -0.56      0.08    -0.72    -0.42 1.00     6366
## beta_ItemLetter1        -1.05      0.07    -1.19    -0.91 1.00    10470
## beta_ItemQueen1         -0.81      0.07    -0.95    -0.67 1.00     7717
## beta_ItemCabin1         -0.87      0.07    -1.01    -0.74 1.00     9133
## beta_ItemPole1           0.34      0.06     0.22     0.46 1.00    14697
## beta_ItemTicket1        -0.48      0.06    -0.61    -0.36 1.00    10560
## beta_ItemGrass1          0.72      0.06     0.60     0.85 1.00    15464
## beta_ItemEngine1         0.73      0.06     0.61     0.85 1.00    13047
## beta_ItemTicket2         1.28      0.08     1.13     1.44 1.00     9791
## beta_ItemCabin2          1.25      0.09     1.07     1.43 1.00     5717
## beta_ItemButter2         0.32      0.07     0.19     0.46 1.00     6806
## beta_ItemShore2          0.31      0.06     0.19     0.43 1.00    11956
## beta_ItemEngine2         1.01      0.08     0.86     1.17 1.00     8287
## beta_ItemArm2            0.36      0.07     0.23     0.49 1.00     8820
## beta_ItemQueen2          1.95      0.09     1.78     2.13 1.00    12117
## beta_ItemLetter2         0.92      0.08     0.77     1.07 1.00     7447
## beta_ItemPole2           2.25      0.10     2.06     2.45 1.00    12961
## beta_ItemGrass2          0.21      0.06     0.10     0.33 1.00    12589
## beta_ItemQueen3          1.41      0.08     1.26     1.56 1.00     9991
## beta_ItemGrass3          1.66      0.10     1.47     1.87 1.00     7176
## beta_ItemArm3            1.20      0.08     1.04     1.36 1.00     7807
## beta_ItemCabin3          1.67      0.08     1.52     1.84 1.00    12315
## beta_ItemPole3           1.83      0.11     1.63     2.04 1.00     7351
## beta_ItemShore3          0.66      0.06     0.53     0.79 1.00     9841
## beta_ItemButter3         1.28      0.07     1.14     1.42 1.00    11118
## beta_ItemEngine3         1.96      0.10     1.77     2.17 1.00    10128
## beta_ItemTicket3         0.94      0.07     0.81     1.07 1.00    10910
## beta_ItemLetter3         0.64      0.06     0.51     0.77 1.00    11960
## logalpha_ItemButter1     0.84      0.21     0.41     1.24 1.00      692
## logalpha_ItemArm1        1.11      0.20     0.70     1.50 1.00      684
## logalpha_ItemShore1      0.82      0.21     0.40     1.21 1.00      664
## logalpha_ItemLetter1     0.06      0.24    -0.42     0.51 1.00      905
## logalpha_ItemQueen1      0.45      0.22     0.00     0.86 1.00      716
## logalpha_ItemCabin1      0.06      0.24    -0.43     0.51 1.00      874
## logalpha_ItemPole1      -0.98      0.38    -1.82    -0.33 1.00     2154
## logalpha_ItemTicket1     0.02      0.24    -0.45     0.48 1.00      859
## logalpha_ItemGrass1     -2.12      0.55    -3.31    -1.16 1.00     5486
## logalpha_ItemEngine1    -2.63      0.53    -3.75    -1.71 1.00     6066
## logalpha_ItemTicket2     0.33      0.23    -0.13     0.75 1.00      812
## logalpha_ItemCabin2      0.82      0.21     0.40     1.21 1.00      689
## logalpha_ItemButter2     0.58      0.21     0.15     0.98 1.00      718
## logalpha_ItemShore2     -0.48      0.30    -1.09     0.06 1.00     1287
## logalpha_ItemEngine2     0.63      0.21     0.22     1.04 1.00      693
## logalpha_ItemArm2        0.36      0.22    -0.09     0.77 1.00      739
## logalpha_ItemQueen2     -1.38      0.54    -2.59    -0.46 1.00     5657
## logalpha_ItemLetter2     0.58      0.21     0.15     0.99 1.00      691
## logalpha_ItemPole2      -1.20      0.55    -2.42    -0.27 1.00     4430
## logalpha_ItemGrass2     -0.83      0.36    -1.62    -0.20 1.00     2134
## logalpha_ItemQueen3      0.16      0.24    -0.34     0.60 1.00      903
## logalpha_ItemGrass3      0.81      0.22     0.38     1.22 1.00      677
## logalpha_ItemArm3        0.61      0.21     0.18     1.02 1.00      714
## logalpha_ItemCabin3     -0.28      0.31    -0.95     0.28 1.00     1353
## logalpha_ItemPole3       0.71      0.22     0.28     1.13 1.00      718
## logalpha_ItemShore3      0.09      0.24    -0.40     0.53 1.00      814
## logalpha_ItemButter3    -0.26      0.28    -0.84     0.27 1.00     1241
## logalpha_ItemEngine3     0.43      0.24    -0.05     0.88 1.00      895
## logalpha_ItemTicket3    -0.11      0.25    -0.63     0.36 1.00     1012
## logalpha_ItemLetter3    -0.20      0.26    -0.74     0.29 1.00     1053
##                      Tail_ESS
## beta_ItemButter1         5403
## beta_ItemArm1            5394
## beta_ItemShore1          5679
## beta_ItemLetter1         6140
## beta_ItemQueen1          5834
## beta_ItemCabin1          5303
## beta_ItemPole1           5038
## beta_ItemTicket1         5474
## beta_ItemGrass1          5777
## beta_ItemEngine1         5191
## beta_ItemTicket2         5960
## beta_ItemCabin2          5771
## beta_ItemButter2         4869
## beta_ItemShore2          4879
## beta_ItemEngine2         6284
## beta_ItemArm2            5991
## beta_ItemQueen2          5197
## beta_ItemLetter2         5662
## beta_ItemPole2           6019
## beta_ItemGrass2          5499
## beta_ItemQueen3          5249
## beta_ItemGrass3          5859
## beta_ItemArm3            5937
## beta_ItemCabin3          5919
## beta_ItemPole3           6353
## beta_ItemShore3          5977
## beta_ItemButter3         6230
## beta_ItemEngine3         6328
## beta_ItemTicket3         6488
## beta_ItemLetter3         5489
## logalpha_ItemButter1     1740
## logalpha_ItemArm1        1717
## logalpha_ItemShore1      1455
## logalpha_ItemLetter1     2594
## logalpha_ItemQueen1      1750
## logalpha_ItemCabin1      2248
## logalpha_ItemPole1       5034
## logalpha_ItemTicket1     2241
## logalpha_ItemGrass1      5571
## logalpha_ItemEngine1     5757
## logalpha_ItemTicket2     1998
## logalpha_ItemCabin2      1598
## logalpha_ItemButter2     1883
## logalpha_ItemShore2      2986
## logalpha_ItemEngine2     1901
## logalpha_ItemArm2        1973
## logalpha_ItemQueen2      5465
## logalpha_ItemLetter2     1801
## logalpha_ItemPole2       4753
## logalpha_ItemGrass2      4091
## logalpha_ItemQueen3      2300
## logalpha_ItemGrass3      1695
## logalpha_ItemArm3        1648
## logalpha_ItemCabin3      3445
## logalpha_ItemPole3       1795
## logalpha_ItemShore3      1938
## logalpha_ItemButter3     3003
## logalpha_ItemEngine3     2120
## logalpha_ItemTicket3     2563
## logalpha_ItemLetter3     3008
## 
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Multidimensional 2PL Model

mcmc_plot(TwoPL_multi, type = "nuts_divergence")

mcmc_plot(TwoPL_multi, type = "nuts_treedepth")

mcmc_plot(TwoPL_multi, type = "trace", variable = "b_beta_Item", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_multi, type = "trace", variable = "b_logalpha_Item", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_multi, type = "trace", variable = "sd_", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_multi, type = "rhat_hist", binwidth = 0.0001)

mcmc_plot(TwoPL_multi, type = "neff_hist", binwidth = 0.1)

The 2PL model with a unique factor per trial shows no significant modeling concerns. Since there are no validity concerns, we can look at the results from the model overall.

mcmc_plot(TwoPL_multi, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta_Item", regex = TRUE)

mcmc_plot(TwoPL_multi, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha_Item", regex = TRUE)

pp_check(TwoPL_multi, ndraws = 50, type = "bars")

pp_check(TwoPL_multi, ndraws = 50, type = "bars_grouped", group = "Item")

pp_check(TwoPL_multi, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))

Like the other 2PL models, this model estimates responses with high accuracy. The following general model summary integrates basic model validity statistics and posterior summaries for additional parameters. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.

summary(TwoPL_multi)
##  Family: bernoulli 
##   Links: mu = logit 
## Formula: Resp ~ beta + exp(logalpha) * theta 
##          theta ~ 0 + (-1 + Trial1 + Trial2 + Trial3 | ID)
##          beta ~ 0 + Item
##          logalpha ~ 0 + Item
##    Data: df_long (Number of observations: 36570) 
##   Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
##          total post-warmup draws = 8000
## 
## Group-Level Effects: 
## ~ID (Number of levels: 1219) 
##                                Estimate Est.Error l-95% CI u-95% CI Rhat
## sd(theta_Trial1)                   0.44      0.15     0.22     0.79 1.00
## sd(theta_Trial2)                   0.53      0.17     0.27     0.92 1.00
## sd(theta_Trial3)                   0.67      0.21     0.35     1.15 1.00
## cor(theta_Trial1,theta_Trial2)     0.99      0.01     0.97     1.00 1.00
## cor(theta_Trial1,theta_Trial3)     0.99      0.01     0.96     1.00 1.00
## cor(theta_Trial2,theta_Trial3)     0.99      0.01     0.98     1.00 1.00
##                                Bulk_ESS Tail_ESS
## sd(theta_Trial1)                   1018     1977
## sd(theta_Trial2)                   1188     1757
## sd(theta_Trial3)                   1171     2298
## cor(theta_Trial1,theta_Trial2)     1016     2169
## cor(theta_Trial1,theta_Trial3)     1158     2405
## cor(theta_Trial2,theta_Trial3)     3625     5411
## 
## Population-Level Effects: 
##                      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_ItemButter1         1.53      0.10     1.34     1.74 1.00     5482
## beta_ItemArm1           -0.10      0.08    -0.26     0.06 1.00     3456
## beta_ItemShore1         -0.57      0.08    -0.71    -0.42 1.00     4222
## beta_ItemLetter1        -1.05      0.07    -1.18    -0.91 1.00     8080
## beta_ItemQueen1         -0.81      0.07    -0.95    -0.67 1.00     7014
## beta_ItemCabin1         -0.87      0.07    -1.00    -0.74 1.00     7749
## beta_ItemPole1           0.34      0.06     0.22     0.46 1.00    11899
## beta_ItemTicket1        -0.48      0.06    -0.61    -0.36 1.00     7791
## beta_ItemGrass1          0.73      0.06     0.61     0.85 1.00    12302
## beta_ItemEngine1         0.73      0.06     0.61     0.85 1.00    11623
## beta_ItemTicket2         1.28      0.08     1.13     1.44 1.00     6665
## beta_ItemCabin2          1.25      0.09     1.07     1.43 1.00     5139
## beta_ItemButter2         0.32      0.07     0.19     0.46 1.00     5936
## beta_ItemShore2          0.31      0.06     0.20     0.43 1.00     8020
## beta_ItemEngine2         1.01      0.08     0.86     1.17 1.00     6205
## beta_ItemArm2            0.36      0.06     0.24     0.49 1.00     6621
## beta_ItemQueen2          1.95      0.09     1.79     2.12 1.00     9987
## beta_ItemLetter2         0.91      0.08     0.77     1.07 1.00     6395
## beta_ItemPole2           2.25      0.10     2.06     2.44 1.00    11506
## beta_ItemGrass2          0.21      0.06     0.10     0.33 1.00    10921
## beta_ItemQueen3          1.41      0.08     1.26     1.56 1.00     7601
## beta_ItemGrass3          1.67      0.10     1.48     1.88 1.00     5803
## beta_ItemArm3            1.20      0.08     1.04     1.36 1.00     6462
## beta_ItemCabin3          1.68      0.08     1.51     1.84 1.00     8818
## beta_ItemPole3           1.83      0.11     1.63     2.05 1.00     6337
## beta_ItemShore3          0.66      0.07     0.53     0.79 1.00     7665
## beta_ItemButter3         1.28      0.07     1.14     1.42 1.00    10672
## beta_ItemEngine3         1.96      0.10     1.76     2.16 1.00     7825
## beta_ItemTicket3         0.94      0.07     0.81     1.08 1.00     9432
## beta_ItemLetter3         0.64      0.06     0.52     0.76 1.00     9960
## logalpha_ItemButter1     1.03      0.34     0.37     1.68 1.00     1089
## logalpha_ItemArm1        1.30      0.33     0.65     1.95 1.00     1081
## logalpha_ItemShore1      1.00      0.33     0.34     1.65 1.00     1035
## logalpha_ItemLetter1     0.24      0.35    -0.45     0.91 1.00     1182
## logalpha_ItemQueen1      0.63      0.34    -0.04     1.28 1.00     1176
## logalpha_ItemCabin1      0.23      0.35    -0.47     0.91 1.00     1169
## logalpha_ItemPole1      -0.84      0.47    -1.84    -0.01 1.00     2309
## logalpha_ItemTicket1     0.20      0.35    -0.49     0.87 1.00     1228
## logalpha_ItemGrass1     -1.99      0.58    -3.25    -0.97 1.00     3697
## logalpha_ItemEngine1    -2.51      0.58    -3.70    -1.44 1.00     4255
## logalpha_ItemTicket2     0.32      0.34    -0.34     0.98 1.00     1323
## logalpha_ItemCabin2      0.81      0.33     0.16     1.46 1.00     1265
## logalpha_ItemButter2     0.57      0.33    -0.06     1.23 1.00     1309
## logalpha_ItemShore2     -0.50      0.38    -1.27     0.24 1.00     1697
## logalpha_ItemEngine2     0.62      0.33    -0.03     1.28 1.00     1257
## logalpha_ItemArm2        0.35      0.34    -0.31     1.01 1.00     1304
## logalpha_ItemQueen2     -1.38      0.57    -2.61    -0.39 1.00     3766
## logalpha_ItemLetter2     0.57      0.34    -0.08     1.24 1.00     1231
## logalpha_ItemPole2      -1.21      0.59    -2.47    -0.18 1.00     3363
## logalpha_ItemGrass2     -0.84      0.43    -1.75    -0.04 1.00     1899
## logalpha_ItemQueen3     -0.08      0.34    -0.74     0.58 1.00     1406
## logalpha_ItemGrass3      0.58      0.32    -0.05     1.20 1.00     1281
## logalpha_ItemArm3        0.38      0.32    -0.24     1.02 1.00     1256
## logalpha_ItemCabin3     -0.51      0.39    -1.30     0.22 1.00     1780
## logalpha_ItemPole3       0.48      0.32    -0.15     1.12 1.00     1268
## logalpha_ItemShore3     -0.15      0.33    -0.80     0.51 1.00     1390
## logalpha_ItemButter3    -0.49      0.37    -1.22     0.21 1.00     1660
## logalpha_ItemEngine3     0.19      0.33    -0.46     0.85 1.00     1361
## logalpha_ItemTicket3    -0.34      0.35    -1.03     0.34 1.00     1406
## logalpha_ItemLetter3    -0.44      0.35    -1.13     0.24 1.00     1474
##                      Tail_ESS
## beta_ItemButter1         5275
## beta_ItemArm1            5032
## beta_ItemShore1          5515
## beta_ItemLetter1         6064
## beta_ItemQueen1          6044
## beta_ItemCabin1          5888
## beta_ItemPole1           5168
## beta_ItemTicket1         5593
## beta_ItemGrass1          6109
## beta_ItemEngine1         5624
## beta_ItemTicket2         5443
## beta_ItemCabin2          5340
## beta_ItemButter2         5903
## beta_ItemShore2          5682
## beta_ItemEngine2         5683
## beta_ItemArm2            5630
## beta_ItemQueen2          5670
## beta_ItemLetter2         6121
## beta_ItemPole2           5996
## beta_ItemGrass2          5785
## beta_ItemQueen3          6006
## beta_ItemGrass3          5862
## beta_ItemArm3            5991
## beta_ItemCabin3          5832
## beta_ItemPole3           6117
## beta_ItemShore3          5955
## beta_ItemButter3         5742
## beta_ItemEngine3         5314
## beta_ItemTicket3         6410
## beta_ItemLetter3         6338
## logalpha_ItemButter1     2285
## logalpha_ItemArm1        2121
## logalpha_ItemShore1      2076
## logalpha_ItemLetter1     2162
## logalpha_ItemQueen1      2214
## logalpha_ItemCabin1      2281
## logalpha_ItemPole1       3893
## logalpha_ItemTicket1     2705
## logalpha_ItemGrass1      4997
## logalpha_ItemEngine1     5001
## logalpha_ItemTicket2     2375
## logalpha_ItemCabin2      2037
## logalpha_ItemButter2     2347
## logalpha_ItemShore2      2660
## logalpha_ItemEngine2     2069
## logalpha_ItemArm2        2035
## logalpha_ItemQueen2      4930
## logalpha_ItemLetter2     1991
## logalpha_ItemPole2       4908
## logalpha_ItemGrass2      3102
## logalpha_ItemQueen3      2761
## logalpha_ItemGrass3      2481
## logalpha_ItemArm3        2313
## logalpha_ItemCabin3      3285
## logalpha_ItemPole3       2301
## logalpha_ItemShore3      3015
## logalpha_ItemButter3     3267
## logalpha_ItemEngine3     2737
## logalpha_ItemTicket3     2989
## logalpha_ItemLetter3     2779
## 
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Latent Change Model

mcmc_plot(TwoPL_chnge, type = "nuts_divergence")

mcmc_plot(TwoPL_chnge, type = "nuts_treedepth")

mcmc_plot(TwoPL_chnge, type = "trace", variable = "b_beta_Item", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_chnge, type = "trace", variable = "b_logalpha_Item", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_chnge, type = "trace", variable = "sd_", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_chnge, type = "rhat_hist", binwidth = 0.0001)

mcmc_plot(TwoPL_chnge, type = "neff_hist", binwidth = 0.1)

The 2PL model with factors for change over each trial shows no significant modeling concerns either. Since there are no validity concerns, we can look at the results from the model overall.

mcmc_plot(TwoPL_chnge, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta_Item", regex = TRUE)

mcmc_plot(TwoPL_chnge, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha_Item", regex = TRUE)

pp_check(TwoPL_chnge, ndraws = 50, type = "bars")

pp_check(TwoPL_chnge, ndraws = 50, type = "bars_grouped", group = "Item")

pp_check(TwoPL_chnge, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))

Consistent with the other 2PL models so far, the model appears to predict the data generation process closely. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.

summary(TwoPL_chnge)
##  Family: bernoulli 
##   Links: mu = logit 
## Formula: Resp ~ beta + exp(logalpha) * theta 
##          theta ~ 0 + (-1 + Time1 + Time2 + Time3 | ID)
##          beta ~ 0 + Item
##          logalpha ~ 0 + Item
##    Data: df_long (Number of observations: 36570) 
##   Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
##          total post-warmup draws = 8000
## 
## Group-Level Effects: 
## ~ID (Number of levels: 1219) 
##                              Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## sd(theta_Time1)                  0.52      0.10     0.35     0.75 1.01      759
## sd(theta_Time2)                  0.05      0.05     0.00     0.19 1.01     1303
## sd(theta_Time3)                  0.05      0.05     0.00     0.18 1.00     1992
## cor(theta_Time1,theta_Time2)     0.12      0.56    -0.90     0.96 1.00     4088
## cor(theta_Time1,theta_Time3)     0.11      0.57    -0.91     0.96 1.00     3986
## cor(theta_Time2,theta_Time3)     0.00      0.51    -0.88     0.90 1.00     5600
##                              Tail_ESS
## sd(theta_Time1)                  1441
## sd(theta_Time2)                  1700
## sd(theta_Time3)                  1756
## cor(theta_Time1,theta_Time2)     2811
## cor(theta_Time1,theta_Time3)     2630
## cor(theta_Time2,theta_Time3)     5242
## 
## Population-Level Effects: 
##                      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_ItemButter1         1.53      0.10     1.33     1.74 1.00     4645
## beta_ItemArm1           -0.10      0.08    -0.25     0.06 1.00     3313
## beta_ItemShore1         -0.57      0.08    -0.72    -0.42 1.00     4745
## beta_ItemLetter1        -1.05      0.07    -1.18    -0.91 1.00     7191
## beta_ItemQueen1         -0.81      0.07    -0.96    -0.68 1.00     7443
## beta_ItemCabin1         -0.87      0.07    -1.01    -0.74 1.00     9514
## beta_ItemPole1           0.34      0.06     0.22     0.46 1.00    13174
## beta_ItemTicket1        -0.48      0.06    -0.61    -0.36 1.00     7845
## beta_ItemGrass1          0.73      0.06     0.61     0.85 1.00    14426
## beta_ItemEngine1         0.73      0.06     0.61     0.85 1.00    12021
## beta_ItemTicket2         1.28      0.08     1.13     1.44 1.00     8259
## beta_ItemCabin2          1.24      0.09     1.07     1.42 1.00     4965
## beta_ItemButter2         0.32      0.07     0.18     0.46 1.00     5622
## beta_ItemShore2          0.31      0.06     0.20     0.43 1.00    12137
## beta_ItemEngine2         1.01      0.08     0.86     1.17 1.00     6019
## beta_ItemArm2            0.36      0.07     0.23     0.49 1.00     6548
## beta_ItemQueen2          1.95      0.09     1.79     2.12 1.00    11104
## beta_ItemLetter2         0.92      0.08     0.77     1.07 1.00     6259
## beta_ItemPole2           2.25      0.10     2.06     2.45 1.00    11038
## beta_ItemGrass2          0.21      0.06     0.10     0.33 1.00    11943
## beta_ItemQueen3          1.41      0.08     1.26     1.56 1.00     8459
## beta_ItemGrass3          1.67      0.10     1.47     1.87 1.00     5593
## beta_ItemArm3            1.19      0.08     1.03     1.36 1.00     6328
## beta_ItemCabin3          1.68      0.08     1.52     1.84 1.00    10753
## beta_ItemPole3           1.83      0.10     1.63     2.04 1.00     6004
## beta_ItemShore3          0.66      0.07     0.53     0.79 1.00     8855
## beta_ItemButter3         1.28      0.07     1.14     1.42 1.00     9180
## beta_ItemEngine3         1.96      0.10     1.76     2.16 1.00     6610
## beta_ItemTicket3         0.94      0.07     0.81     1.07 1.00     8500
## beta_ItemLetter3         0.64      0.06     0.52     0.76 1.00     8680
## logalpha_ItemButter1     0.81      0.22     0.39     1.26 1.01      940
## logalpha_ItemArm1        1.08      0.21     0.68     1.50 1.01      876
## logalpha_ItemShore1      0.79      0.22     0.37     1.21 1.01      876
## logalpha_ItemLetter1     0.03      0.25    -0.46     0.51 1.00     1153
## logalpha_ItemQueen1      0.42      0.23    -0.02     0.85 1.01      966
## logalpha_ItemCabin1      0.02      0.24    -0.46     0.50 1.01     1193
## logalpha_ItemPole1      -1.02      0.40    -1.94    -0.34 1.00     2604
## logalpha_ItemTicket1    -0.01      0.24    -0.49     0.46 1.00     1110
## logalpha_ItemGrass1     -2.13      0.54    -3.32    -1.21 1.00     5225
## logalpha_ItemEngine1    -2.65      0.55    -3.80    -1.68 1.00     5299
## logalpha_ItemTicket2     0.27      0.23    -0.17     0.70 1.01     1099
## logalpha_ItemCabin2      0.76      0.21     0.34     1.16 1.01      890
## logalpha_ItemButter2     0.52      0.21     0.10     0.93 1.01      920
## logalpha_ItemShore2     -0.54      0.29    -1.16     0.00 1.00     1798
## logalpha_ItemEngine2     0.58      0.21     0.15     0.98 1.01      940
## logalpha_ItemArm2        0.30      0.22    -0.15     0.70 1.01     1016
## logalpha_ItemQueen2     -1.41      0.54    -2.66    -0.52 1.00     5032
## logalpha_ItemLetter2     0.52      0.21     0.10     0.93 1.01      934
## logalpha_ItemPole2      -1.23      0.53    -2.42    -0.35 1.00     5040
## logalpha_ItemGrass2     -0.88      0.35    -1.64    -0.29 1.00     2159
## logalpha_ItemQueen3      0.07      0.25    -0.44     0.54 1.01     1168
## logalpha_ItemGrass3      0.73      0.22     0.30     1.15 1.01      913
## logalpha_ItemArm3        0.53      0.22     0.09     0.95 1.01      952
## logalpha_ItemCabin3     -0.36      0.31    -1.02     0.21 1.01     1740
## logalpha_ItemPole3       0.63      0.22     0.20     1.06 1.01      954
## logalpha_ItemShore3      0.00      0.24    -0.48     0.47 1.01     1122
## logalpha_ItemButter3    -0.34      0.29    -0.93     0.19 1.01     1544
## logalpha_ItemEngine3     0.34      0.24    -0.15     0.79 1.01     1092
## logalpha_ItemTicket3    -0.19      0.26    -0.72     0.31 1.01     1355
## logalpha_ItemLetter3    -0.28      0.26    -0.83     0.21 1.01     1362
##                      Tail_ESS
## beta_ItemButter1         5414
## beta_ItemArm1            5023
## beta_ItemShore1          5481
## beta_ItemLetter1         5506
## beta_ItemQueen1          5691
## beta_ItemCabin1          6228
## beta_ItemPole1           5650
## beta_ItemTicket1         5974
## beta_ItemGrass1          6436
## beta_ItemEngine1         6461
## beta_ItemTicket2         5512
## beta_ItemCabin2          5753
## beta_ItemButter2         4808
## beta_ItemShore2          5526
## beta_ItemEngine2         5845
## beta_ItemArm2            5604
## beta_ItemQueen2          6067
## beta_ItemLetter2         5806
## beta_ItemPole2           6427
## beta_ItemGrass2          5513
## beta_ItemQueen3          6367
## beta_ItemGrass3          5832
## beta_ItemArm3            6040
## beta_ItemCabin3          6162
## beta_ItemPole3           5857
## beta_ItemShore3          5778
## beta_ItemButter3         6054
## beta_ItemEngine3         6254
## beta_ItemTicket3         5800
## beta_ItemLetter3         5448
## logalpha_ItemButter1     2225
## logalpha_ItemArm1        1823
## logalpha_ItemShore1      1910
## logalpha_ItemLetter1     2501
## logalpha_ItemQueen1      2104
## logalpha_ItemCabin1      2580
## logalpha_ItemPole1       4265
## logalpha_ItemTicket1     2472
## logalpha_ItemGrass1      5416
## logalpha_ItemEngine1     5200
## logalpha_ItemTicket2     2312
## logalpha_ItemCabin2      1952
## logalpha_ItemButter2     1967
## logalpha_ItemShore2      3391
## logalpha_ItemEngine2     2140
## logalpha_ItemArm2        2382
## logalpha_ItemQueen2      4633
## logalpha_ItemLetter2     2295
## logalpha_ItemPole2       5094
## logalpha_ItemGrass2      4639
## logalpha_ItemQueen3      2501
## logalpha_ItemGrass3      2258
## logalpha_ItemArm3        2362
## logalpha_ItemCabin3      3316
## logalpha_ItemPole3       2573
## logalpha_ItemShore3      2407
## logalpha_ItemButter3     3394
## logalpha_ItemEngine3     2586
## logalpha_ItemTicket3     3242
## logalpha_ItemLetter3     3007
## 
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Latent Growth (Learning) Model

mcmc_plot(TwoPL_learn, type = "nuts_divergence")

mcmc_plot(TwoPL_learn, type = "nuts_treedepth")

mcmc_plot(TwoPL_learn, type = "trace", variable = "b_beta_Item", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_learn, type = "trace", variable = "b_logalpha_Item", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_learn, type = "trace", variable = "sd_", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_learn, type = "rhat_hist", binwidth = 0.0001)

mcmc_plot(TwoPL_learn, type = "neff_hist", binwidth = 0.1)

The 2PL model with a growth/learning factor also yields no estimation errors. Since there are no validity concerns for the model, we can look at the results from the model.

mcmc_plot(TwoPL_learn, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta_Item", regex = TRUE)

mcmc_plot(TwoPL_learn, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha_Item", regex = TRUE)

pp_check(TwoPL_learn, ndraws = 50, type = "bars")

pp_check(TwoPL_learn, ndraws = 50, type = "bars_grouped", group = "Item")

pp_check(TwoPL_learn, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))

Continuing the trends of the other models, these posterior checks closely align with the observed data. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.

summary(TwoPL_learn)
##  Family: bernoulli 
##   Links: mu = logit 
## Formula: Resp ~ beta + exp(logalpha) * theta 
##          theta ~ 0 + (1 + Time.n | ID) + (-1 + Trial1 | ID) + (-1 + Trial2 | ID) + (-1 + Trial3 | ID)
##          beta ~ 0 + Item
##          logalpha ~ 0 + Item
##    Data: df_long (Number of observations: 36570) 
##   Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
##          total post-warmup draws = 8000
## 
## Group-Level Effects: 
## ~ID (Number of levels: 1219) 
##                                   Estimate Est.Error l-95% CI u-95% CI Rhat
## sd(theta_Intercept)                   0.55      0.12     0.32     0.81 1.01
## sd(theta_Time.n)                      0.05      0.06     0.00     0.22 1.00
## sd(theta_Trial1)                      0.03      0.03     0.00     0.10 1.00
## sd(theta_Trial2)                      0.02      0.02     0.00     0.07 1.00
## sd(theta_Trial3)                      0.03      0.03     0.00     0.10 1.00
## cor(theta_Intercept,theta_Time.n)     0.24      0.65    -0.96     0.99 1.00
##                                   Bulk_ESS Tail_ESS
## sd(theta_Intercept)                    729     1030
## sd(theta_Time.n)                       900     1179
## sd(theta_Trial1)                      1721     3222
## sd(theta_Trial2)                      3787     4578
## sd(theta_Trial3)                      2885     3484
## cor(theta_Intercept,theta_Time.n)     1943     2871
## 
## Population-Level Effects: 
##                      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_ItemButter1         1.53      0.10     1.34     1.74 1.00     6054
## beta_ItemArm1           -0.10      0.08    -0.26     0.06 1.00     4385
## beta_ItemShore1         -0.57      0.08    -0.72    -0.42 1.00     5129
## beta_ItemLetter1        -1.05      0.07    -1.18    -0.91 1.00     9892
## beta_ItemQueen1         -0.81      0.07    -0.95    -0.68 1.00     8400
## beta_ItemCabin1         -0.87      0.07    -1.01    -0.74 1.00    12392
## beta_ItemPole1           0.34      0.06     0.22     0.46 1.00    13341
## beta_ItemTicket1        -0.48      0.06    -0.61    -0.36 1.00    10550
## beta_ItemGrass1          0.72      0.06     0.60     0.85 1.00    15931
## beta_ItemEngine1         0.73      0.06     0.61     0.85 1.00    17977
## beta_ItemTicket2         1.28      0.08     1.13     1.44 1.00     9431
## beta_ItemCabin2          1.24      0.09     1.07     1.42 1.00     5939
## beta_ItemButter2         0.32      0.07     0.18     0.45 1.00     7000
## beta_ItemShore2          0.31      0.06     0.19     0.43 1.00    13969
## beta_ItemEngine2         1.01      0.08     0.86     1.17 1.00     7397
## beta_ItemArm2            0.36      0.07     0.23     0.49 1.00     8191
## beta_ItemQueen2          1.95      0.09     1.78     2.13 1.00    15205
## beta_ItemLetter2         0.91      0.07     0.77     1.06 1.00     6864
## beta_ItemPole2           2.25      0.10     2.06     2.45 1.00    16402
## beta_ItemGrass2          0.21      0.06     0.10     0.33 1.00    14702
## beta_ItemQueen3          1.41      0.08     1.25     1.57 1.00    10286
## beta_ItemGrass3          1.66      0.10     1.47     1.88 1.00     6546
## beta_ItemArm3            1.20      0.08     1.03     1.36 1.00     6909
## beta_ItemCabin3          1.68      0.08     1.52     1.84 1.00    11822
## beta_ItemPole3           1.83      0.11     1.63     2.04 1.00     6641
## beta_ItemShore3          0.66      0.07     0.53     0.79 1.00    10202
## beta_ItemButter3         1.28      0.07     1.14     1.42 1.00    11481
## beta_ItemEngine3         1.96      0.10     1.77     2.16 1.00     8372
## beta_ItemTicket3         0.94      0.07     0.81     1.07 1.00    11121
## beta_ItemLetter3         0.64      0.06     0.52     0.76 1.00    11140
## logalpha_ItemButter1     0.77      0.25     0.31     1.31 1.00      865
## logalpha_ItemArm1        1.04      0.25     0.60     1.57 1.01      819
## logalpha_ItemShore1      0.75      0.25     0.29     1.27 1.00      836
## logalpha_ItemLetter1    -0.01      0.27    -0.54     0.55 1.00     1049
## logalpha_ItemQueen1      0.38      0.25    -0.10     0.92 1.00      899
## logalpha_ItemCabin1     -0.01      0.27    -0.54     0.54 1.00      996
## logalpha_ItemPole1      -1.05      0.41    -1.95    -0.33 1.00     2110
## logalpha_ItemTicket1    -0.05      0.27    -0.57     0.52 1.00      980
## logalpha_ItemGrass1     -2.17      0.54    -3.33    -1.23 1.00     4892
## logalpha_ItemEngine1    -2.70      0.55    -3.87    -1.69 1.00     5602
## logalpha_ItemTicket2     0.20      0.23    -0.27     0.65 1.00      942
## logalpha_ItemCabin2      0.69      0.21     0.28     1.11 1.00      839
## logalpha_ItemButter2     0.45      0.21     0.04     0.88 1.00      846
## logalpha_ItemShore2     -0.60      0.29    -1.23    -0.07 1.00     1507
## logalpha_ItemEngine2     0.51      0.21     0.09     0.92 1.00      858
## logalpha_ItemArm2        0.23      0.22    -0.19     0.65 1.00      892
## logalpha_ItemQueen2     -1.46      0.54    -2.70    -0.56 1.00     4673
## logalpha_ItemLetter2     0.45      0.22     0.03     0.87 1.00      854
## logalpha_ItemPole2      -1.28      0.52    -2.49    -0.42 1.00     4148
## logalpha_ItemGrass2     -0.94      0.35    -1.73    -0.34 1.00     2307
## logalpha_ItemQueen3     -0.01      0.26    -0.55     0.48 1.00     1021
## logalpha_ItemGrass3      0.64      0.24     0.16     1.10 1.00      774
## logalpha_ItemArm3        0.44      0.24    -0.05     0.91 1.00      833
## logalpha_ItemCabin3     -0.44      0.33    -1.15     0.14 1.00     1490
## logalpha_ItemPole3       0.55      0.24     0.06     1.00 1.00      790
## logalpha_ItemShore3     -0.08      0.26    -0.59     0.40 1.00      934
## logalpha_ItemButter3    -0.42      0.30    -1.05     0.14 1.00     1241
## logalpha_ItemEngine3     0.25      0.26    -0.29     0.74 1.00      957
## logalpha_ItemTicket3    -0.27      0.28    -0.84     0.25 1.00     1053
## logalpha_ItemLetter3    -0.37      0.28    -0.94     0.16 1.00     1196
##                      Tail_ESS
## beta_ItemButter1         5753
## beta_ItemArm1            5361
## beta_ItemShore1          5686
## beta_ItemLetter1         6420
## beta_ItemQueen1          6089
## beta_ItemCabin1          5795
## beta_ItemPole1           5727
## beta_ItemTicket1         5657
## beta_ItemGrass1          5732
## beta_ItemEngine1         5627
## beta_ItemTicket2         5657
## beta_ItemCabin2          5659
## beta_ItemButter2         5289
## beta_ItemShore2          5468
## beta_ItemEngine2         5708
## beta_ItemArm2            5316
## beta_ItemQueen2          5656
## beta_ItemLetter2         6187
## beta_ItemPole2           5756
## beta_ItemGrass2          5306
## beta_ItemQueen3          5577
## beta_ItemGrass3          6185
## beta_ItemArm3            6322
## beta_ItemCabin3          6117
## beta_ItemPole3           6204
## beta_ItemShore3          5276
## beta_ItemButter3         5670
## beta_ItemEngine3         5910
## beta_ItemTicket3         5275
## beta_ItemLetter3         5728
## logalpha_ItemButter1     1317
## logalpha_ItemArm1        1186
## logalpha_ItemShore1      1244
## logalpha_ItemLetter1     1671
## logalpha_ItemQueen1      1279
## logalpha_ItemCabin1      1553
## logalpha_ItemPole1       2657
## logalpha_ItemTicket1     1435
## logalpha_ItemGrass1      5177
## logalpha_ItemEngine1     5197
## logalpha_ItemTicket2     2128
## logalpha_ItemCabin2      1821
## logalpha_ItemButter2     1867
## logalpha_ItemShore2      3244
## logalpha_ItemEngine2     1883
## logalpha_ItemArm2        2364
## logalpha_ItemQueen2      5111
## logalpha_ItemLetter2     1937
## logalpha_ItemPole2       4795
## logalpha_ItemGrass2      3924
## logalpha_ItemQueen3      2516
## logalpha_ItemGrass3      2006
## logalpha_ItemArm3        2051
## logalpha_ItemCabin3      2853
## logalpha_ItemPole3       1828
## logalpha_ItemShore3      2556
## logalpha_ItemButter3     2939
## logalpha_ItemEngine3     2295
## logalpha_ItemTicket3     2586
## logalpha_ItemLetter3     2813
## 
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Local Dependency Model

mcmc_plot(TwoPL_depmd, type = "nuts_divergence")

mcmc_plot(TwoPL_depmd, type = "nuts_treedepth")

mcmc_plot(TwoPL_depmd, type = "trace", variable = "b_beta_Item", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_depmd, type = "trace", variable = "b_logalpha_Item", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_depmd, type = "trace", variable = "sd_", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_depmd, type = "rhat_hist", binwidth = 0.0001)

mcmc_plot(TwoPL_depmd, type = "neff_hist", binwidth = 0.1)

The local dependency effect on the 2PL model fit without any modeling concerns. Given the lack of validity issues, the results from the model overall are shown next.

mcmc_plot(TwoPL_depmd, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta_Item", regex = TRUE)

mcmc_plot(TwoPL_depmd, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha_Item", regex = TRUE)

pp_check(TwoPL_depmd, ndraws = 50, type = "bars")

pp_check(TwoPL_depmd, ndraws = 50, type = "bars_grouped", group = "Item")

pp_check(TwoPL_depmd, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))

As with the other models, the predictive checks closely align to the data. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.

summary(TwoPL_depmd)
##  Family: bernoulli 
##   Links: mu = logit 
## Formula: Resp ~ beta + exp(logalpha) * theta 
##          theta ~ 0 + (1 | ID)
##          beta ~ 0 + LocDep + Item
##          logalpha ~ 0 + LocDep + Item
##    Data: df_long (Number of observations: 36570) 
##   Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
##          total post-warmup draws = 8000
## 
## Group-Level Effects: 
## ~ID (Number of levels: 1219) 
##                     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(theta_Intercept)     0.41      0.08     0.27     0.58 1.00      935     1800
## 
## Population-Level Effects: 
##                      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_LocDep              0.60      0.04     0.53     0.68 1.00     5052
## beta_ItemButter1         1.52      0.10     1.34     1.72 1.00     6091
## beta_ItemArm1           -0.10      0.08    -0.26     0.06 1.00     5202
## beta_ItemShore1         -0.56      0.08    -0.71    -0.41 1.00     6770
## beta_ItemLetter1        -1.04      0.07    -1.18    -0.91 1.00     9319
## beta_ItemQueen1         -0.80      0.07    -0.94    -0.67 1.00     7982
## beta_ItemCabin1         -0.86      0.07    -0.99    -0.73 1.00     9718
## beta_ItemPole1           0.34      0.06     0.23     0.45 1.00    11492
## beta_ItemTicket1        -0.48      0.06    -0.60    -0.36 1.00    10765
## beta_ItemGrass1          0.73      0.06     0.61     0.85 1.00    11512
## beta_ItemEngine1         0.73      0.06     0.61     0.85 1.00    12922
## beta_ItemTicket2         0.79      0.08     0.64     0.95 1.00     7419
## beta_ItemCabin2          0.92      0.09     0.75     1.10 1.00     6568
## beta_ItemButter2         0.07      0.07    -0.06     0.20 1.00     6842
## beta_ItemShore2          0.15      0.06     0.04     0.27 1.00    10930
## beta_ItemEngine2         0.81      0.08     0.66     0.97 1.00     8139
## beta_ItemArm2            0.17      0.06     0.05     0.30 1.00     8543
## beta_ItemQueen2          1.63      0.09     1.46     1.81 1.00     9460
## beta_ItemLetter2         0.69      0.08     0.54     0.84 1.00     7856
## beta_ItemPole2           1.88      0.10     1.68     2.08 1.00    11310
## beta_ItemGrass2         -0.19      0.06    -0.32    -0.07 1.00     9334
## beta_ItemQueen3          0.99      0.08     0.83     1.16 1.00     7728
## beta_ItemGrass3          1.33      0.11     1.13     1.55 1.00     5763
## beta_ItemArm3            0.73      0.09     0.57     0.91 1.00     6118
## beta_ItemCabin3          1.34      0.08     1.18     1.51 1.00     9223
## beta_ItemPole3           1.55      0.11     1.35     1.77 1.00     7472
## beta_ItemShore3          0.20      0.07     0.07     0.34 1.00     7356
## beta_ItemButter3         0.85      0.08     0.71     1.00 1.00     8224
## beta_ItemEngine3         1.65      0.11     1.45     1.87 1.00     8041
## beta_ItemTicket3         0.42      0.07     0.28     0.57 1.00     7823
## beta_ItemLetter3         0.10      0.07    -0.04     0.24 1.00     7396
## logalpha_LocDep          0.21      0.08     0.06     0.36 1.00     6010
## logalpha_ItemButter1     1.06      0.22     0.62     1.49 1.00     1246
## logalpha_ItemArm1        1.33      0.22     0.90     1.76 1.00     1124
## logalpha_ItemShore1      1.02      0.22     0.59     1.44 1.00     1182
## logalpha_ItemLetter1     0.24      0.25    -0.27     0.72 1.00     1504
## logalpha_ItemQueen1      0.64      0.23     0.20     1.08 1.00     1333
## logalpha_ItemCabin1      0.18      0.26    -0.34     0.67 1.00     1451
## logalpha_ItemPole1      -0.88      0.44    -1.88    -0.16 1.00     3640
## logalpha_ItemTicket1     0.20      0.25    -0.31     0.69 1.00     1392
## logalpha_ItemGrass1     -2.00      0.58    -3.28    -1.00 1.00     5679
## logalpha_ItemEngine1    -2.50      0.56    -3.70    -1.49 1.00     6114
## logalpha_ItemTicket2     0.10      0.27    -0.44     0.60 1.00     1721
## logalpha_ItemCabin2      0.75      0.23     0.29     1.21 1.00     1340
## logalpha_ItemButter2     0.50      0.23     0.03     0.94 1.00     1331
## logalpha_ItemShore2     -0.71      0.38    -1.56    -0.04 1.00     2864
## logalpha_ItemEngine2     0.65      0.23     0.20     1.09 1.00     1316
## logalpha_ItemArm2        0.34      0.23    -0.13     0.80 1.00     1326
## logalpha_ItemQueen2     -1.43      0.57    -2.64    -0.46 1.00     6123
## logalpha_ItemLetter2     0.67      0.23     0.23     1.11 1.00     1134
## logalpha_ItemPole2      -1.17      0.57    -2.45    -0.21 1.00     5408
## logalpha_ItemGrass2     -0.75      0.38    -1.58    -0.10 1.00     2799
## logalpha_ItemQueen3     -0.01      0.29    -0.60     0.52 1.00     2059
## logalpha_ItemGrass3      0.87      0.23     0.43     1.32 1.00     1218
## logalpha_ItemArm3        0.61      0.23     0.16     1.06 1.00     1267
## logalpha_ItemCabin3     -0.76      0.47    -1.84     0.01 1.00     3541
## logalpha_ItemPole3       0.86      0.22     0.43     1.30 1.00     1225
## logalpha_ItemShore3     -0.15      0.28    -0.75     0.37 1.00     1672
## logalpha_ItemButter3    -0.63      0.40    -1.53     0.05 1.00     2712
## logalpha_ItemEngine3     0.54      0.25     0.05     1.01 1.00     1338
## logalpha_ItemTicket3    -0.05      0.27    -0.59     0.45 1.00     1744
## logalpha_ItemLetter3    -0.21      0.28    -0.78     0.31 1.00     1738
##                      Tail_ESS
## beta_LocDep              6083
## beta_ItemButter1         5669
## beta_ItemArm1            5179
## beta_ItemShore1          5106
## beta_ItemLetter1         6352
## beta_ItemQueen1          5888
## beta_ItemCabin1          6218
## beta_ItemPole1           6070
## beta_ItemTicket1         5935
## beta_ItemGrass1          5620
## beta_ItemEngine1         5695
## beta_ItemTicket2         5932
## beta_ItemCabin2          5477
## beta_ItemButter2         5276
## beta_ItemShore2          5271
## beta_ItemEngine2         5757
## beta_ItemArm2            5844
## beta_ItemQueen2          5528
## beta_ItemLetter2         5739
## beta_ItemPole2           5564
## beta_ItemGrass2          5824
## beta_ItemQueen3          5210
## beta_ItemGrass3          5420
## beta_ItemArm3            6124
## beta_ItemCabin3          6130
## beta_ItemPole3           6112
## beta_ItemShore3          6219
## beta_ItemButter3         5757
## beta_ItemEngine3         5856
## beta_ItemTicket3         5887
## beta_ItemLetter3         5773
## logalpha_LocDep          6098
## logalpha_ItemButter1     2585
## logalpha_ItemArm1        2320
## logalpha_ItemShore1      2367
## logalpha_ItemLetter1     3319
## logalpha_ItemQueen1      2818
## logalpha_ItemCabin1      3219
## logalpha_ItemPole1       4464
## logalpha_ItemTicket1     3203
## logalpha_ItemGrass1      3779
## logalpha_ItemEngine1     5411
## logalpha_ItemTicket2     3662
## logalpha_ItemCabin2      2742
## logalpha_ItemButter2     2979
## logalpha_ItemShore2      4071
## logalpha_ItemEngine2     2735
## logalpha_ItemArm2        2678
## logalpha_ItemQueen2      5345
## logalpha_ItemLetter2     2673
## logalpha_ItemPole2       4571
## logalpha_ItemGrass2      4210
## logalpha_ItemQueen3      3764
## logalpha_ItemGrass3      2440
## logalpha_ItemArm3        2617
## logalpha_ItemCabin3      4601
## logalpha_ItemPole3       2642
## logalpha_ItemShore3      2949
## logalpha_ItemButter3     4151
## logalpha_ItemEngine3     2649
## logalpha_ItemTicket3     3657
## logalpha_ItemLetter3     3641
## 
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Local Dependency by Item Model

mcmc_plot(TwoPL_depun, type = "nuts_divergence")

mcmc_plot(TwoPL_depun, type = "nuts_treedepth")
## Warning: Groups with fewer than two data points have been dropped.

## Warning: Groups with fewer than two data points have been dropped.

mcmc_plot(TwoPL_depun, type = "trace", variable = "b_beta_Item", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_depun, type = "trace", variable = "b_logalpha_Item", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_depun, type = "trace", variable = "sd_", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_depun, type = "rhat_hist", binwidth = 0.0001)

mcmc_plot(TwoPL_depun, type = "neff_hist", binwidth = 0.1)

As with the model with a uniform local dependency effect, the model with a unique dependency effect per item demonstrated no estimation errors, so next is shown the results from the model overall.

mcmc_plot(TwoPL_depun, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta_Item", regex = TRUE)

mcmc_plot(TwoPL_depun, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha_Item", regex = TRUE)

pp_check(TwoPL_depun, ndraws = 50, type = "bars")

pp_check(TwoPL_depun, ndraws = 50, type = "bars_grouped", group = "Item")

pp_check(TwoPL_depun, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))

These posterior predictive checks also strongly align with the observed data. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.

summary(TwoPL_depun)
##  Family: bernoulli 
##   Links: mu = logit 
## Formula: Resp ~ beta + exp(logalpha) * theta 
##          theta ~ 0 + (1 | ID)
##          beta ~ 0 + DepButter + DepArm + DepShore + DepLetter + DepQueen + DepCabin + DepPole + DepTicket + DepGrass + DepEngine + Item
##          logalpha ~ 0 + DepButter + DepArm + DepShore + DepLetter + DepQueen + DepCabin + DepPole + DepTicket + DepGrass + DepEngine + Item
##    Data: df_long (Number of observations: 36570) 
##   Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
##          total post-warmup draws = 8000
## 
## Group-Level Effects: 
## ~ID (Number of levels: 1219) 
##                     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(theta_Intercept)     0.41      0.08     0.27     0.60 1.00     1656     3063
## 
## Population-Level Effects: 
##                      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_DepButter           0.42      0.15     0.13     0.70 1.00     6081
## beta_DepArm              0.61      0.14     0.33     0.88 1.00     6854
## beta_DepShore            0.72      0.12     0.49     0.94 1.00     8988
## beta_DepLetter           0.97      0.11     0.76     1.19 1.00    11481
## beta_DepQueen            0.50      0.13     0.24     0.76 1.00     9653
## beta_DepCabin            0.45      0.11     0.24     0.66 1.00    11129
## beta_DepPole             0.60      0.12     0.37     0.83 1.00    11237
## beta_DepTicket           0.60      0.13     0.36     0.85 1.00    11049
## beta_DepGrass            0.64      0.14     0.37     0.90 1.00    11050
## beta_DepEngine           0.55      0.11     0.33     0.76 1.00    10940
## beta_ItemButter1         1.53      0.10     1.35     1.74 1.00     9297
## beta_ItemArm1           -0.10      0.08    -0.27     0.06 1.00     7705
## beta_ItemShore1         -0.56      0.08    -0.71    -0.42 1.00     8562
## beta_ItemLetter1        -1.04      0.07    -1.17    -0.90 1.00    13880
## beta_ItemQueen1         -0.81      0.07    -0.95    -0.67 1.00    12415
## beta_ItemCabin1         -0.87      0.07    -1.00    -0.73 1.00    14653
## beta_ItemPole1           0.34      0.06     0.22     0.45 1.00    16442
## beta_ItemTicket1        -0.48      0.06    -0.60    -0.36 1.00    13703
## beta_ItemGrass1          0.73      0.06     0.60     0.85 1.00    16950
## beta_ItemEngine1         0.73      0.06     0.61     0.85 1.00    17010
## beta_ItemTicket2         0.95      0.16     0.65     1.27 1.00     5671
## beta_ItemCabin2          0.88      0.11     0.66     1.11 1.00     6654
## beta_ItemButter2         0.02      0.08    -0.13     0.18 1.00     8007
## beta_ItemShore2          0.07      0.06    -0.05     0.20 1.00    14322
## beta_ItemEngine2         0.85      0.09     0.67     1.03 1.00     8126
## beta_ItemArm2            0.22      0.07     0.08     0.36 1.00     9590
## beta_ItemQueen2          1.64      0.10     1.44     1.84 1.00    12186
## beta_ItemLetter2         0.70      0.09     0.53     0.88 1.00     8609
## beta_ItemPole2           1.86      0.12     1.62     2.11 1.00     9995
## beta_ItemGrass2         -0.16      0.09    -0.34     0.03 1.00    10008
## beta_ItemQueen3          1.12      0.13     0.87     1.39 1.00     5995
## beta_ItemGrass3          1.32      0.12     1.09     1.57 1.00     7801
## beta_ItemArm3            0.64      0.12     0.42     0.88 1.00     7561
## beta_ItemCabin3          1.19      0.09     1.01     1.38 1.00    12009
## beta_ItemPole3           1.59      0.12     1.36     1.83 1.00     9393
## beta_ItemShore3          0.32      0.10     0.12     0.52 1.00    10480
## beta_ItemButter3         0.85      0.11     0.65     1.07 1.00    11056
## beta_ItemEngine3         1.65      0.12     1.42     1.89 1.00    10334
## beta_ItemTicket3         0.40      0.13     0.14     0.66 1.00    10781
## beta_ItemLetter3         0.15      0.12    -0.08     0.37 1.00     9981
## logalpha_DepButter       0.03      0.33    -0.56     0.73 1.00     5525
## logalpha_DepArm          0.45      0.19     0.08     0.83 1.00     6001
## logalpha_DepShore        0.24      0.19    -0.13     0.63 1.00     6948
## logalpha_DepLetter      -0.58      0.67    -2.04     0.58 1.00    11320
## logalpha_DepQueen        0.13      0.17    -0.21     0.47 1.00     8643
## logalpha_DepCabin        0.07      0.22    -0.37     0.49 1.00     9773
## logalpha_DepPole         0.34      0.54    -0.81     1.33 1.00     9536
## logalpha_DepTicket       0.10      0.19    -0.28     0.46 1.00     9028
## logalpha_DepGrass        0.29      0.40    -0.43     1.14 1.00     9816
## logalpha_DepEngine      -0.09      0.38    -0.79     0.71 1.00     8309
## logalpha_ItemButter1     1.06      0.22     0.63     1.50 1.00     1987
## logalpha_ItemArm1        1.33      0.22     0.89     1.75 1.00     1940
## logalpha_ItemShore1      0.99      0.22     0.55     1.42 1.00     2050
## logalpha_ItemLetter1     0.20      0.26    -0.33     0.68 1.00     2674
## logalpha_ItemQueen1      0.64      0.23     0.20     1.07 1.00     2251
## logalpha_ItemCabin1      0.20      0.25    -0.33     0.69 1.00     2641
## logalpha_ItemPole1      -0.91      0.43    -1.90    -0.18 1.00     5389
## logalpha_ItemTicket1     0.19      0.25    -0.32     0.67 1.00     2197
## logalpha_ItemGrass1     -2.01      0.57    -3.24    -1.02 1.00     9930
## logalpha_ItemEngine1    -2.51      0.55    -3.69    -1.51 1.00     8689
## logalpha_ItemTicket2     0.29      0.38    -0.52     0.96 1.00     3783
## logalpha_ItemCabin2      0.61      0.27     0.07     1.13 1.00     2511
## logalpha_ItemButter2     0.40      0.27    -0.16     0.91 1.00     2582
## logalpha_ItemShore2     -0.85      0.46    -1.90    -0.09 1.00     5712
## logalpha_ItemEngine2     0.69      0.24     0.21     1.16 1.00     2535
## logalpha_ItemArm2        0.43      0.26    -0.09     0.91 1.00     2527
## logalpha_ItemQueen2     -1.52      0.58    -2.76    -0.52 1.00     8274
## logalpha_ItemLetter2     0.70      0.24     0.22     1.17 1.00     2229
## logalpha_ItemPole2      -1.29      0.62    -2.60    -0.18 1.00     6796
## logalpha_ItemGrass2     -0.53      0.48    -1.57     0.30 1.00     4745
## logalpha_ItemQueen3      0.13      0.32    -0.54     0.73 1.00     3473
## logalpha_ItemGrass3      0.74      0.25     0.25     1.23 1.00     2349
## logalpha_ItemArm3        0.55      0.26     0.02     1.04 1.00     2336
## logalpha_ItemCabin3     -0.96      0.56    -2.22    -0.02 1.00     6693
## logalpha_ItemPole3       0.86      0.24     0.39     1.32 1.00     2307
## logalpha_ItemShore3     -0.01      0.30    -0.62     0.56 1.00     3414
## logalpha_ItemButter3    -0.78      0.47    -1.77     0.04 1.00     5995
## logalpha_ItemEngine3     0.56      0.26     0.05     1.07 1.00     2321
## logalpha_ItemTicket3    -0.14      0.40    -0.99     0.60 1.00     5791
## logalpha_ItemLetter3    -0.02      0.38    -0.82     0.68 1.00     5003
##                      Tail_ESS
## beta_DepButter           6305
## beta_DepArm              6434
## beta_DepShore            6605
## beta_DepLetter           6392
## beta_DepQueen            6960
## beta_DepCabin            6571
## beta_DepPole             6405
## beta_DepTicket           6888
## beta_DepGrass            5483
## beta_DepEngine           6014
## beta_ItemButter1         6675
## beta_ItemArm1            6290
## beta_ItemShore1          6256
## beta_ItemLetter1         6005
## beta_ItemQueen1          6313
## beta_ItemCabin1          6020
## beta_ItemPole1           5714
## beta_ItemTicket1         5669
## beta_ItemGrass1          5472
## beta_ItemEngine1         5420
## beta_ItemTicket2         6668
## beta_ItemCabin2          6647
## beta_ItemButter2         6183
## beta_ItemShore2          6317
## beta_ItemEngine2         6954
## beta_ItemArm2            6496
## beta_ItemQueen2          6594
## beta_ItemLetter2         6312
## beta_ItemPole2           6735
## beta_ItemGrass2          6506
## beta_ItemQueen3          6031
## beta_ItemGrass3          6436
## beta_ItemArm3            6478
## beta_ItemCabin3          6690
## beta_ItemPole3           6721
## beta_ItemShore3          6785
## beta_ItemButter3         6243
## beta_ItemEngine3         6963
## beta_ItemTicket3         6561
## beta_ItemLetter3         6386
## logalpha_DepButter       5674
## logalpha_DepArm          6063
## logalpha_DepShore        5732
## logalpha_DepLetter       6450
## logalpha_DepQueen        6350
## logalpha_DepCabin        6439
## logalpha_DepPole         5927
## logalpha_DepTicket       6617
## logalpha_DepGrass        6399
## logalpha_DepEngine       6109
## logalpha_ItemButter1     3694
## logalpha_ItemArm1        3262
## logalpha_ItemShore1      3709
## logalpha_ItemLetter1     4215
## logalpha_ItemQueen1      4204
## logalpha_ItemCabin1      4290
## logalpha_ItemPole1       5529
## logalpha_ItemTicket1     4127
## logalpha_ItemGrass1      5348
## logalpha_ItemEngine1     5090
## logalpha_ItemTicket2     4954
## logalpha_ItemCabin2      3596
## logalpha_ItemButter2     4209
## logalpha_ItemShore2      5933
## logalpha_ItemEngine2     4592
## logalpha_ItemArm2        4178
## logalpha_ItemQueen2      6244
## logalpha_ItemLetter2     4316
## logalpha_ItemPole2       6412
## logalpha_ItemGrass2      5358
## logalpha_ItemQueen3      5186
## logalpha_ItemGrass3      4325
## logalpha_ItemArm3        4227
## logalpha_ItemCabin3      5820
## logalpha_ItemPole3       4163
## logalpha_ItemShore3      5113
## logalpha_ItemButter3     6520
## logalpha_ItemEngine3     4258
## logalpha_ItemTicket3     5683
## logalpha_ItemLetter3     6095
## 
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Local Dependency by Trial Model

mcmc_plot(TwoPL_deptr, type = "nuts_divergence")

mcmc_plot(TwoPL_deptr, type = "nuts_treedepth")

mcmc_plot(TwoPL_deptr, type = "trace", variable = "b_beta_Item", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_deptr, type = "trace", variable = "b_logalpha_Item", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_deptr, type = "trace", variable = "sd_", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_deptr, type = "rhat_hist", binwidth = 0.0001)

mcmc_plot(TwoPL_deptr, type = "neff_hist", binwidth = 0.1)

Decomposing the dependency matrix into unique trial effects, rather than unique item effects, also does not cause any modeling validity concerns. As with the other models, the overall results are shown next.

mcmc_plot(TwoPL_deptr, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta_Item", regex = TRUE)

mcmc_plot(TwoPL_deptr, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha_Item", regex = TRUE)

pp_check(TwoPL_deptr, ndraws = 50, type = "bars")

pp_check(TwoPL_deptr, ndraws = 50, type = "bars_grouped", group = "Item")

pp_check(TwoPL_deptr, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))

Posterior predictive checks are consistent with the other models in that the model predictions and observed data closely align. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.

summary(TwoPL_deptr)
##  Family: bernoulli 
##   Links: mu = logit 
## Formula: Resp ~ beta + exp(logalpha) * theta 
##          theta ~ 0 + (1 | ID)
##          beta ~ 0 + LocDep.1 + LocDep.2 + Item
##          logalpha ~ 0 + LocDep.1 + LocDep.2 + Item
##    Data: df_long (Number of observations: 36570) 
##   Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
##          total post-warmup draws = 8000
## 
## Group-Level Effects: 
## ~ID (Number of levels: 1219) 
##                     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(theta_Intercept)     0.47      0.09     0.31     0.67 1.00      781     1188
## 
## Population-Level Effects: 
##                      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_LocDep.1            1.03      0.05     0.93     1.13 1.00     5496
## beta_LocDep.2            0.12      0.06     0.00     0.24 1.00     2727
## beta_ItemButter1         1.51      0.10     1.33     1.72 1.00     6278
## beta_ItemArm1           -0.10      0.08    -0.26     0.06 1.00     5321
## beta_ItemShore1         -0.56      0.08    -0.71    -0.41 1.00     6364
## beta_ItemLetter1        -1.05      0.07    -1.19    -0.91 1.00    10297
## beta_ItemQueen1         -0.80      0.07    -0.94    -0.67 1.00     7502
## beta_ItemCabin1         -0.87      0.07    -1.00    -0.74 1.00     9728
## beta_ItemPole1           0.34      0.06     0.22     0.46 1.00    11433
## beta_ItemTicket1        -0.48      0.06    -0.60    -0.36 1.00    10139
## beta_ItemGrass1          0.73      0.06     0.61     0.85 1.00    11444
## beta_ItemEngine1         0.73      0.06     0.61     0.85 1.00    11609
## beta_ItemTicket2         0.51      0.08     0.35     0.68 1.00     6772
## beta_ItemCabin2          0.80      0.09     0.63     0.98 1.00     5746
## beta_ItemButter2        -0.01      0.07    -0.15     0.13 1.00     6227
## beta_ItemShore2          0.07      0.06    -0.05     0.19 1.00     9760
## beta_ItemEngine2         0.76      0.08     0.60     0.92 1.00     6575
## beta_ItemArm2            0.09      0.07    -0.05     0.23 1.00     7097
## beta_ItemQueen2          1.45      0.09     1.27     1.64 1.00     9284
## beta_ItemLetter2         0.58      0.08     0.43     0.73 1.00     6801
## beta_ItemPole2           1.65      0.10     1.46     1.86 1.00     9731
## beta_ItemGrass2         -0.49      0.07    -0.63    -0.36 1.00     7036
## beta_ItemQueen3          1.30      0.09     1.13     1.48 1.00     4605
## beta_ItemGrass3          1.56      0.11     1.35     1.78 1.00     4352
## beta_ItemArm3            1.08      0.09     0.90     1.27 1.00     4051
## beta_ItemCabin3          1.59      0.09     1.42     1.76 1.00     5495
## beta_ItemPole3           1.77      0.11     1.56     2.00 1.00     5384
## beta_ItemShore3          0.55      0.08     0.40     0.70 1.00     4006
## beta_ItemButter3         1.17      0.08     1.01     1.34 1.00     4982
## beta_ItemEngine3         1.87      0.11     1.67     2.08 1.00     6741
## beta_ItemTicket3         0.83      0.09     0.67     1.00 1.00     4622
## beta_ItemLetter3         0.53      0.08     0.36     0.69 1.00     4170
## logalpha_LocDep.1       -0.57      0.17    -0.94    -0.26 1.00     5977
## logalpha_LocDep.2        0.28      0.11     0.07     0.49 1.00     3280
## logalpha_ItemButter1     0.90      0.23     0.45     1.34 1.00      928
## logalpha_ItemArm1        1.12      0.22     0.69     1.55 1.00      925
## logalpha_ItemShore1      0.85      0.22     0.42     1.28 1.00      999
## logalpha_ItemLetter1     0.18      0.25    -0.31     0.65 1.00     1303
## logalpha_ItemQueen1      0.49      0.23     0.04     0.94 1.00     1056
## logalpha_ItemCabin1      0.06      0.25    -0.45     0.55 1.00     1369
## logalpha_ItemPole1      -0.88      0.40    -1.77    -0.20 1.00     2551
## logalpha_ItemTicket1     0.03      0.25    -0.49     0.53 1.00     1304
## logalpha_ItemGrass1     -1.96      0.56    -3.18    -1.01 1.00     6534
## logalpha_ItemEngine1    -2.56      0.53    -3.67    -1.61 1.00     5711
## logalpha_ItemTicket2     0.06      0.32    -0.62     0.64 1.00     1832
## logalpha_ItemCabin2      0.56      0.24     0.09     1.04 1.00     1170
## logalpha_ItemButter2     0.55      0.24     0.06     1.01 1.00     1276
## logalpha_ItemShore2     -0.68      0.40    -1.55    -0.00 1.00     2651
## logalpha_ItemEngine2     0.68      0.23     0.21     1.13 1.00     1049
## logalpha_ItemArm2        0.50      0.24     0.03     0.96 1.00     1098
## logalpha_ItemQueen2     -1.44      0.60    -2.74    -0.44 1.00     5757
## logalpha_ItemLetter2     0.68      0.24     0.21     1.13 1.00     1069
## logalpha_ItemPole2      -0.90      0.57    -2.19     0.05 1.00     6077
## logalpha_ItemGrass2      0.06      0.31    -0.60     0.62 1.00     1849
## logalpha_ItemQueen3      0.01      0.26    -0.52     0.51 1.00     1487
## logalpha_ItemGrass3      0.72      0.23     0.28     1.17 1.00     1094
## logalpha_ItemArm3        0.47      0.23     0.00     0.92 1.00     1235
## logalpha_ItemCabin3     -0.46      0.34    -1.19     0.16 1.00     2653
## logalpha_ItemPole3       0.67      0.23     0.22     1.12 1.00     1083
## logalpha_ItemShore3     -0.01      0.25    -0.50     0.47 1.00     1288
## logalpha_ItemButter3    -0.46      0.32    -1.14     0.15 1.00     2162
## logalpha_ItemEngine3     0.32      0.25    -0.19     0.79 1.00     1296
## logalpha_ItemTicket3    -0.28      0.27    -0.82     0.23 1.00     1621
## logalpha_ItemLetter3    -0.31      0.26    -0.85     0.19 1.00     1611
##                      Tail_ESS
## beta_LocDep.1            6268
## beta_LocDep.2            3966
## beta_ItemButter1         6424
## beta_ItemArm1            5368
## beta_ItemShore1          5635
## beta_ItemLetter1         6349
## beta_ItemQueen1          5282
## beta_ItemCabin1          5895
## beta_ItemPole1           5968
## beta_ItemTicket1         5828
## beta_ItemGrass1          5962
## beta_ItemEngine1         5980
## beta_ItemTicket2         6252
## beta_ItemCabin2          5981
## beta_ItemButter2         6355
## beta_ItemShore2          5227
## beta_ItemEngine2         5841
## beta_ItemArm2            5167
## beta_ItemQueen2          5713
## beta_ItemLetter2         5898
## beta_ItemPole2           5614
## beta_ItemGrass2          6343
## beta_ItemQueen3          5262
## beta_ItemGrass3          4964
## beta_ItemArm3            5280
## beta_ItemCabin3          5066
## beta_ItemPole3           6364
## beta_ItemShore3          5432
## beta_ItemButter3         5792
## beta_ItemEngine3         6407
## beta_ItemTicket3         5931
## beta_ItemLetter3         4896
## logalpha_LocDep.1        5430
## logalpha_LocDep.2        4819
## logalpha_ItemButter1     1579
## logalpha_ItemArm1        1319
## logalpha_ItemShore1      1852
## logalpha_ItemLetter1     3075
## logalpha_ItemQueen1      2364
## logalpha_ItemCabin1      2752
## logalpha_ItemPole1       3917
## logalpha_ItemTicket1     2704
## logalpha_ItemGrass1      5089
## logalpha_ItemEngine1     5260
## logalpha_ItemTicket2     3250
## logalpha_ItemCabin2      2403
## logalpha_ItemButter2     2714
## logalpha_ItemShore2      4175
## logalpha_ItemEngine2     2169
## logalpha_ItemArm2        2457
## logalpha_ItemQueen2      5519
## logalpha_ItemLetter2     2346
## logalpha_ItemPole2       5342
## logalpha_ItemGrass2      3961
## logalpha_ItemQueen3      3382
## logalpha_ItemGrass3      2150
## logalpha_ItemArm3        2540
## logalpha_ItemCabin3      4431
## logalpha_ItemPole3       2528
## logalpha_ItemShore3      2422
## logalpha_ItemButter3     4092
## logalpha_ItemEngine3     2654
## logalpha_ItemTicket3     3381
## logalpha_ItemLetter3     3432
## 
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Item Covariates

mcmc_plot(TwoPL_itmex, type = "nuts_divergence")

mcmc_plot(TwoPL_itmex, type = "nuts_treedepth")

mcmc_plot(TwoPL_itmex, type = "trace", variable = "b_beta", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_itmex, type = "trace", variable = "b_logalpha", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_itmex, type = "trace", variable = "sd_", regex = TRUE)
## No divergences to plot.

mcmc_plot(TwoPL_itmex, type = "rhat_hist", binwidth = 0.0001)

mcmc_plot(TwoPL_itmex, type = "neff_hist", binwidth = 0.1)

The 2PL model with all item covariates demonstrated no evidence of estimation concerns that would raise concerns for the validity of the results. Since there are no validity concerns, we can look at the results from the model overall.

mcmc_plot(TwoPL_itmex, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_beta", regex = TRUE)

mcmc_plot(TwoPL_itmex, type = "areas_ridges", prob = 0.80, prob_outer = 0.95, variable = "b_logalpha", regex = TRUE)

pp_check(TwoPL_itmex, ndraws = 50, type = "bars")

pp_check(TwoPL_itmex, ndraws = 50, type = "bars_grouped", group = "ID", newdata = subset(df_long, df_long$ID %in% as.factor(sample.int(n = 1219, size = 20, replace = FALSE))))

Like the other 2PL models, this model estimates responses with high accuracy. The following general model summary integrates basic model validity statistics and posterior summaries for additional parameters. See the guide at the end of the Rasch intercept tab for details regarding the meaning of each value.

summary(TwoPL_itmex)
##  Family: bernoulli 
##   Links: mu = logit 
## Formula: Resp ~ beta + exp(logalpha) * theta 
##          theta ~ 0 + (1 | ID)
##          beta ~ 1 + Time.c + poly(ItemPos, 2) + FreqSTX + Concrete + Diversity + AoA + BOI + Phonemes
##          logalpha ~ 1 + Time.c + poly(ItemPos, 2) + FreqSTX + Concrete + Diversity + AoA + BOI + Phonemes
##    Data: df_long (Number of observations: 36570) 
##   Draws: 4 chains, each with iter = 3000; warmup = 1000; thin = 1;
##          total post-warmup draws = 8000
## 
## Group-Level Effects: 
## ~ID (Number of levels: 1219) 
##                     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(theta_Intercept)     0.75      0.47     0.13     1.91 1.00    12602     4763
## 
## Population-Level Effects: 
##                        Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## beta_Intercept            -0.04      0.02    -0.09     0.00 1.00     6323
## beta_Time.c1               0.97      0.03     0.91     1.03 1.00    12657
## beta_Time.c2               1.34      0.03     1.28     1.40 1.00    13518
## beta_polyItemPos21        12.46      1.59     9.27    15.58 1.00    21398
## beta_polyItemPos22        29.94      1.53    26.95    33.00 1.00    18744
## beta_FreqSTX              -0.35      0.02    -0.38    -0.31 1.00    11432
## beta_Concrete             -0.40      0.02    -0.45    -0.35 1.00     8659
## beta_Diversity            -0.21      0.02    -0.26    -0.17 1.00     8776
## beta_AoA                  -0.17      0.02    -0.21    -0.13 1.00    14061
## beta_BOI                   0.37      0.03     0.32     0.42 1.00     7587
## beta_Phonemes              0.11      0.02     0.08     0.15 1.00    10479
## logalpha_Intercept        -0.22      0.70    -1.38     1.32 1.00    12706
## logalpha_Time.c1           0.22      0.06     0.10     0.35 1.00     9325
## logalpha_Time.c2           0.21      0.07     0.08     0.34 1.00     9057
## logalpha_polyItemPos21    -2.00      0.98    -3.92    -0.09 1.00    20742
## logalpha_polyItemPos22    -0.93      1.00    -2.89     1.02 1.00    18711
## logalpha_FreqSTX           0.16      0.04     0.08     0.24 1.00     8919
## logalpha_Concrete          0.06      0.06    -0.05     0.17 1.00     7797
## logalpha_Diversity        -0.14      0.06    -0.26    -0.03 1.00     7449
## logalpha_AoA               0.10      0.05    -0.01     0.20 1.00    10494
## logalpha_BOI               0.19      0.05     0.09     0.29 1.00     7155
## logalpha_Phonemes         -0.25      0.05    -0.34    -0.16 1.00     7099
##                        Tail_ESS
## beta_Intercept             6393
## beta_Time.c1               6739
## beta_Time.c2               6991
## beta_polyItemPos21         5766
## beta_polyItemPos22         5982
## beta_FreqSTX               6509
## beta_Concrete              6580
## beta_Diversity             6804
## beta_AoA                   6318
## beta_BOI                   6778
## beta_Phonemes              6825
## logalpha_Intercept         5044
## logalpha_Time.c1           7306
## logalpha_Time.c2           6336
## logalpha_polyItemPos21     5633
## logalpha_polyItemPos22     5620
## logalpha_FreqSTX           6709
## logalpha_Concrete          6695
## logalpha_Diversity         6553
## logalpha_AoA               6053
## logalpha_BOI               6850
## logalpha_Phonemes          6658
## 
## Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Final Model Details

This final section provides further details on the final item covariate model identified in this study. To begin with, some additional details regarding the model’s performance and fit are provided for readers. The manuscript provides summaries of coefficients as well as item and person fit statistics. In a previous section, this supplemental manuscript provided visual summaries of the coefficients as well. To further enrich the visual understanding of the final data, the following plots describe item and person fit further:

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

These two plots show the empirical distribution of the differences in log-likelihood of the predicted and observed data under the models. The first plot is grouped by items and reflects the item fit. In an ideal model, the distribution is normally distributed with a mean of 0. To address the impracticality of plotting all of these plots for each of 1219 participants, just the most extreme case (individual with worst fit) is shown. While the aim of this study was to examine predictors of item traits, it is still possible to visualize person traits (i.e., memory). The following plot shows the ordered distribution of latent trait estimates and their 95% credible intervals for each participant in the study:

The plots demonstrate that, on the whole, the estimates of a person’s latent trait is fairly wide. This is not entirely surprising given that the only estimate being given to the latent trait is a random intercept for each person. The explanatory item response theory model will be expanded in a future study to include factors that predict latent trait, which may reduce uncertainty in the latent trait measurement. The largest limiter to higher accuracy in the latent trait estimate is the fact that memory is being measured with just 10 items repeated only 3 times, so person predictors are likely not going to be a significant source of error reduction. It is also potentially useful to visualize all the conditional effects from the final model. These plots are shown next:

There are also some general test descriptives that are usually helpful to examine. These include test reliability, the expected score function, and test information. The following plots are these visualizations in their respective order:

## `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'

## `summarise()` has grouped output by 'Theta'. You can override using the `.groups` argument.
## `summarise()` has grouped output by 'Theta'. You can override using the `.groups` argument.
## `summarise()` has grouped output by 'Theta', 'Trial'. You can override using the `.groups` argument.

## `summarise()` has grouped output by 'Theta'. You can override using the `.groups` argument.
## `summarise()` has grouped output by 'Theta'. You can override using the `.groups` argument.

## `summarise()` has grouped output by 'Theta'. You can override using the `.groups` argument.
## `summarise()` has grouped output by 'Theta'. You can override using the `.groups` argument.

Within IRT, information and reliability are very closely related. It is clear that these plots both have generally the same shape with peaks in their functions around slightly below average scores. As is clear from both of these scales, the CERAD word list is not ideal for measuring individuals with either very low or very high memory abilities. In dementia/mild cognitive impairment research, this particular trait may be favorable as the goal is often to characterize with greater precision mildly impaired memory with relatively little need for carefully measuring extremes. The expected score function is a useful visual analogue for the relationship between raw score and latent trait estimate. The relationship is clearly monotonic and overall very gradual, which means that the raw scores do span a relatively wide range of latent trait values.